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ABSTRACT 


There  is  an  obvious  need  for  performance  measurement  in  U.  S.  Navy 
commands.  Not  only  are  performance  metrics,  or  measurements,  essential  to 
tracking  progress  toward  strategic  goals,  but  as  a  publicly  funded  entity  and 
holder  of  the  public  trust,  the  Navy  has  an  obligation  to  efficiently  and  effectively 
use  those  public  funds.  Performance  metrics  are  the  guideposts  to  achieving 
efficiency  and  effectiveness.  This  thesis  was  designed  to  aid  the  Program 
Executive  Office  for  Integrated  Warfare  Systems,  a  Navy  Echelon  II  acquisition 
command,  with  a  performance  measurement  project,  deriving  metrics  for  two  of 
its  seven  major  programs,  IWS  1.0  and  2.0.  Performance  drivers  were  captured 
through  interviews  with  key  leaders  in  those  two  major  programs.  Those 
interviews  were  transformed  into  causal  performance  maps  which  depicted  the 
interplay  of  the  drivers  and  the  outcomes  they  influenced.  Performance  metrics 
were  then  derived  for  those  drivers  and  outcomes  and  arranged  in  a  balanced 
scorecard  format.  The  scorecards  may  be  useful  to  the  major  program 
managers  in  monitoring  the  progress  of  their  organizations  toward  achieving 
strategic  success.  Additionally,  the  metrics  should  enhance  understanding  of 
strategic  direction  by  the  rank  and  file  of  IWS  1 .0  and  2.0. 
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I.  INTRODUCTION 


A.  BACKGROUND 

Performance  measurement  came  to  the  forefront  of  federal  agency 
consciousness  with  the  passage  of  the  Government  Performance  and  Results 
Act  of  1993  by  the  United  States  Congress.  President  Bush  and  President 
Clinton  before  him  both  emphasized  improved  efficiency  and  effectiveness  in  the 
federal  government  through  strategic  planning  and  performance  measurement. 
Controlling  and  measuring  performance  in  the  attainment  of  strategic  objectives 
is  an  essential  part  of  the  Navy’s  plan  to  improve  its  capabilities  over  the  long 
run. 


Program  Executive  Office  for  Integrated  Warfare  Systems  (PEO  IWS)  is  a 
Navy  Echelon  III  command  involved  in  the  acquisition  of  shipboard  weapon 
systems.  The  primary  role  of  PEO  IWS  personnel  is  to  shepherd  and  manage 
weapon  systems  through  the  programming  and  acquisition  process;  they  do  not 
physically  produce  the  weapon  systems  themselves.  As  such  PEO  IWS 
depends  almost  wholly  on  the  experience  and  capability  of  its  manpower;  it  is  a 
knowledge  based  organization. 

The  command  was  established  in  2003  in  order  to 

...Provide  the  required  discipline  and  coordination  of  the 
architecture  and  overarching  interface  principles  to  which  our 
systems  will  be  developed.  This  PEO  will  be  responsible  for  all 
surface  ship  and  submarine  combat  systems,  missiles  (except 
TRIDENT  and  TOMAHAWK),  radars,  launchers  (except  TRIDENT), 

EW  and  gun  systems.  This  PEO  will  combine  the  combat  systems 
software  programs  from  the  present  PEO  Subs,  Carriers,  TSC, 

EXW  and  Surface  Strike.  This  PEO  will  also  be  responsible  for 
coordinating  all  ASW  warfare  area  programs  across  PEOs.  ASW 
software  development,  including  the  annual  submarine  combat 
systems  software  update,  will  migrate  to  PEO  IWS,  facilitating 
optimal  development  of  ASW  software  across  platforms.  PEO 
Subs  will  continue  as  the  complimentary  buy  and  build  organization 
for  the  supporting  components  and  torpedoes. 

This  realignment  changes  our  focus  from  the  current  platform 
centered  approach  to  a  more  integrated  approach  across  all 
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combat  systems.  As  the  Navy  moves  to  open  systems 
architectures  and  highly  integrated  systems  of  systems,  it  is  critical 
that  those  efforts  have  a  strong,  consistent  focus. 

(Deputy  Assistant  Secretary  of  the  Navy  2003) 

The  command’s  creation  was  in  response  to  the  Chief  of  Naval 
Operations’  vision  for  optimal  alignment  of  Navy  organizations  in  order  to 
successfully  implement  his  strategy  embodied  in  SEA  POWER  21.  PEO  IWS 
was  faced  with  the  real  challenge  of  combining  entities  from  five  different 
commands,  with  five  different  cultures,  into  an  effective  single  organization.  This 
required  the  creation  of  an  organizational  structure,  infrastructure,  processes, 
and  systems  for  achieving  its  mission.  Another  part  of  that  organizational  design 
challenge  was  creating  a  useful  performance  measurement  system  that  could  be 
used  to  indicate  their  progress  toward  strategic  goals. 


PEO  IWS  was  subsequently  organized  around  seven  major  program 
areas.  These  major  programs,  known  as  IWS  1.0  through  IWS  7.0  are  listed 
below. 


IWS  1.0 

Integrated  Combat  Systems 

IWS  2.0 

Above  Water  Sensors 

IWS  3.0 

Surface  Ship  Weapons 

IWS  4.0 

International  Programs 

IWS  5.0 

Undersea  Systems 

IWS  6.0 

Command  and  Control  (Networks/Excomm) 

IWS  7.0 

Naval  Open  Architecture 

Performance  measurement  is  a  crucial  tool  to  the  major  program 
managers  within  PEO  IWS.  If  implemented  properly,  it  has  the  potential  to 
answer  two  questions  of  significant  importance: 


•  Are  we  doing  the  right  things? 

•  Are  we  doing  the  right  things  well? 
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B.  OBJECTIVE 


The  purpose  of  this  thesis  was  to  observe  strategy,  goals  and  operations 
at  PEO  IWS  1 .0  and  2.0  with  the  aim  of  recommending  performance  metrics  that 
might  be  useful  for  the  major  program  managers  in  determining  how  well  their 
organizations  are  meeting  their  strategic  objectives.  IWS  1.0  and  2.0  were 
studied,  and  after  analysis  of  collected  data,  recommended  performance  metrics 
were  suggested. 

C.  RESEARCH  QUESTION 

1.  Primary  Research  Question 

•  What  are  useful  performance  metrics  for  PEO  IWS  1 .0  and  2.0? 

2.  Secondary  Research  Questions 

•  What  are  the  vision,  mission,  and  strategy  of  PEO  IWS  and  its 
chain  of  command? 

•  What  current  performance  metrics  are  being  utilized  by  IWS  1.0 
and  IWS  2.0? 

•  What  guidance  does  the  Navy  leadership  give  regarding 
performance  measurement? 

D.  SCOPE  AND  LIMITATIONS 

1.  PEO  IWS  1.0  and  2.0 

This  thesis  is  limited  to  the  two  aforementioned  major  program  areas  at 
PEO  IWS.  These  two  major  programs  may  have  much  in  common  with  other 
major  programs  at  IWS  and  within  the  Navy,  but  the  thrust  of  this  effort  was  to 
evaluate  these  two  entities  only. 

2.  Strategic  Goals 

The  recommendations  herein  are  limited  to  measures  of  effectiveness  and 
efficiency  in  achieving  strategic  goals.  It  is  possible  to  drill  down  to  lower  and 
lower  levels  of  an  organization,  and  create  more  specific  and  tactically  oriented 
metrics.  Measures  are  limited  to  those  appropriate  for  the  major  program 
manager  level  at  IWS  1 .0  and  2.0. 
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3.  Perspective 

While  there  are  many  individuals  and  organizations  with  a  stake  in  the 
outcome  of  the  work  performed  by  PEO  IWS  1 .0  and  2.0,  this  thesis  was  written 
from  the  perspective  that  its  primary  users  would  be  the  major  program 
managers  of  those  two  units. 

4.  Limitations 

Limitations  imposed  by  distance  have  made  close  interaction  more  difficult 
than  if  the  author  had  been  collocated  with  PEO  IWS.  Additionally,  the  author’s 
limited  experience  with  Navy  acquisition  undoubtedly  will  surface  occasionally  in 
this  thesis. 

E.  METHODOLOGY 

This  thesis  attempted  to  derive  useful  performance  metrics  by  leveraging 
the  personal  knowledge  of  IWS  1.0  and  2.0  leaders  and  managers,  as  well  as 
leading  literature  in  the  field  of  performance  measurement. 

1.  Tacit  Knowledge 

There  are  different  sources  for  deriving  what  is  important  to  the  success  of 
an  organization.  The  primary  source  chosen  for  this  thesis  was  to  tap  into 
organizational  managers’  tacit  knowledge,  or  that  knowledge  known  to  the 
individual,  but  not  usually  recorded  anywhere.  Individuals  may  or  may  not  realize 
the  depth  of  their  expertise,  or  the  intricacy  to  which  they  understand  their  own 
organizations.  Individuals’  tacit  knowledge,  based  primarily  on  experience  and 
education,  can  provide  a  significant  source  of  information  about  the 
organization’s  performance,  particularly  in  knowledge-based  organizations  like 
PEO  IWS  (Ambrosini  and  Bowman  2002). 

2.  Interviews 

Interviews  were  used  to  extract  tacit  knowledge  from  leadership  within 
IWS  1.0  and  2.0.  In  accord  with  Spradley’s  The  Ethnographic  Interview  (1979), 
questioning  was  conducted  as  informally  as  possible,  with  the  hope  of  building  a 
rapport  with  interview  subjects  so  that  maximum  cooperation  might  be  obtained. 
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The  ultimate  goal  was  to  allow  interview  subjects  to  express  what  they  saw  as 
the  drivers  of  performance  within  IWS  1.0  and  2.0  through  observations,  stories, 
and  anecdotes. 

3.  Causal  Performance  Maps 

The  performance  drivers  and  linkages  among  them  were  identified  in  each 
interview.  Interview  data  were  then  translated  into  cognitive  maps  known  as 
causal  performance  maps.  A  cognitive  map  is  a  representation  of  an  individual’s 
knowledge,  experience,  and  point  of  view  (Ambrosini  and  Bowman  2002).  A 
causal  map  attempts  to  visually  portray  the  cause  and  effect  relationships  among 
performance  factors  that  affect  a  desired  outcome. 

4.  Performance  Metrics 

After  causal  performance  maps  were  created,  reviewed,  and  edited  by 
interview  subjects,  critical  performance  variables  were  identified  and  metrics 
were  derived  based  on  past  research  and  performance  measurement  literature. 
Additionally,  interview  subjects  were  asked  to  give  their  recommendations  for 
useful  metrics. 

F.  ORGANIZATION  OF  THESIS 

Following  this  introduction,  Chapter  II  reviews  leading  research  in  the  field 
of  performance  measurement.  Chapter  III  presents  collected  data  in  the  form  of 
PEO  IWS  and  Navy-wide  strategic  objectives  and  vision  and  causal  performance 
maps.  An  analysis  of  the  collected  data  is  offered  in  Chapter  IV,  Chapter  V 
presents  recommendations  and  summary. 
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II.  LITERATURE  REVIEW 


A.  WHY  BE  CONCERNED  WITH  PERFORMANCE  MEASUEMENT? 

1.  Government  Performance  and  Results  Act  of  1993  (GPRA) 

In  January  1993,  Congress  passed  GPRA  in  order  to: 

(1)  improve  the  confidence  of  the  American  people  in  the 
capability  of  the  Federal  Government,  by  systematically 
holding  Federal  agencies  accountable  for  achieving  program 
results; 

(2)  initiate  program  performance  reform  with  a  series  of  pilot 
projects  in  setting  program  goals,  measuring  program 
performance  against  those  goals,  and  reporting  publicly  on 
their  progress; 

(3)  improve  Federal  program  effectiveness  and  public 
accountability  by  promoting  a  new  focus  on  results,  service 
quality,  and  customer  satisfaction; 

(4)  help  Federal  managers  improve  service  delivery,  by 
requiring  that  they  plan  for  meeting  program  objectives  and  by 
providing  them  with  information  about  program  results  and 
service  quality; 

(5)  improve  congressional  decision-making  by  providing  more 
objective  information  on  achieving  statutory  objectives,  and  on 
the  relative  effectiveness  and  efficiency  of  Federal  programs 
and  spending;  and 

(6)  improve  internal  management  of  the  Federal  Government. 
(Government  Performance  and  Results  Act  1993) 

Additionally,  GPRA  required  that  federal  agencies  incorporate  and  report  to 

Congress  the  results  of  their  strategic  planning,  including  a  mission  statement, 

outcome-related  goals  and  objectives,  an  analysis  of  how  the  goals  and 

objectives  would  be  achieved,  and  a  description  of  how  performance  goals  relate 

to  strategic  goals  and  objectives.  Since  public  organizations  do  not  operate  to 

maximize  profits,  much  of  the  efficiency  derived  from  competitive  markets  was 

generally  not  being  observed  in  federal  agencies.  Congress  was  attempting  to 

improve  efficiency  and  effectiveness  by  mandating  practices  that  were  common 

in  private-sector  enterprises,  while  also  increasing  the  accountability  of  public 
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agencies.  Performance  measurement  against  strategic  goals  and  objectives  was 
deemed  a  key  ingredient  in  improving  public  agency  performance. 

2.  President  Bush’s  Management  Agenda 

In  his  fiscal  year  (FY)  2002  Management  Agenda,  President  Bush 
articulated  management  improvements  that  he  sought  to  execute  throughout  the 
federal  government.  These  improvements  were  to  be  “results  oriented”  and  to 
“achieve  immediate,  concrete,  and  measurable  results.”  Additionally,  it  was 
stated  that  subsequent  goals  would  be  undertaken  as  “tangible  improvements 
are  made.”  The  President  planned  to  formally  integrate  performance  review  with 
budget  decisions,  stipulate  that  agencies  pinpoint  outcome  measures  and 
accurately  monitor  the  progress  of  their  programs,  and  that  the  entire  Executive 
Branch  management  become  more  performance-oriented.  It  was  very  evident 
that  the  President’s  agenda  could  not  be  achieved  without  robust  and  accurate 
performance  measurement. 

3.  Department  of  the  Navy 

Performance  measurement  is  a  significant  tool  in  the  Secretary  of  the 
Navy’s  plan  for  improving  efficiency  and  effectiveness  within  the  Navy.  It  was  a 
major  theme  in  the  Secretary’s  FY  2006/2007  budget  submission  to  the 
Secretary  of  Defense,  comprising  nearly  one-tenth  of  the  total  verbiage.  In  his 
“Principles  of  Leadership,”  the  Secretary  encourages  Navy  leaders  to 
“incorporate  measures  and  metrics  everywhere.”  (England  2003). 

Likewise,  the  Chief  of  Naval  Operations  specifically  mentioned  metrics 
thirteen  different  times  in  his  “Guidance  for  2005,”  his  specific  vision  and  goals 
disseminated  to  the  entire  Navy.  The  implication  is  that  the  Navy  has  specific 
strategic  goals,  and  it  cannot  tell  how  well  it  is  achieving  those  goals  without 
specifically  measuring  its  performance. 


B.  CYBERNETIC  FEEDBACK  MODEL 

One  of  the  most  basic  models  of  organizational  function  is  the  inputs- 
process-outputs  model.  In  this  model,  organizational  functions  are  broken  down 
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into  inputs  to  the  process,  processes  that  perform  some  transformation  of  the 
inputs,  and  outputs  that  are  the  result  of  the  transformation.  The  organization 
exists  to  produce  outputs.  The  leaders  and  members  of  the  organization  direct 
their  efforts  to  the  transformation  of  inputs  for  the  delivery  of  outputs. 


Inputs 


Process 


► 


Outputs 


Figure  1.  Inputs-Process-Outputs  Model  (From:  Simons  2000) 


A  significant  improvement  on  this  model  is  the  cybernetic  feedback  model. 
The  cybernetic  feedback  model  incorporates  a  standard  or  expectation  for  the 
output  and  a  feedback  loop.  The  standard  allows  the  organization  to  define  an 
acceptable  output  and  the  feedback  loop  indicates  to  leadership  whether  the 
processes  are  producing  outputs  that  meet  the  predefined  standard.  This 
comparison  of  output  versus  an  expectation  is  the  hallmark  of  performance 
measurement  systems.  Variance  from  the  standard  is  fed  back  to  members  and 
leaders  of  the  organization  so  that  they  can  alter  inputs  and  processes  in  an 
attempt  to  better  meet  the  standard.  Thus,  the  goal  of  performance 
measurement  is  to  direct  and  control  organizational  energy  toward  a  desired 
outcome  through  systematic  measurement. 

Standard 

i 

*  (^)  - ►  Outputs 

♦  ♦ 


Feedback 


Inputs 


Figure  2.  Cybernetic  Feedback  Model  (From:  Simons  2000) 
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C.  THE  NATURE  OF  PERFORMANCE  METRICS 

Modern  researchers  (Kaplan  and  Norton  1996,  Lynch  and  Cross  1995, 
Simons  2000,  Malina  and  Selto  2001)  have  taken  a  detailed  look  at  performance 
measurement  systems,  both  their  derivations  and  characteristics,  over  the  last 
three  decades.  Some  of  the  common  themes  that  emerge  from  the  research  of 
the  last  decade  include: 

1 .  Performance  metrics  should  be  linked  to  strategy. 

2.  Performance  metrics  should  be  focused  on  what  is  important  to  the 

success  of  the  organization 

3.  Effective  metrics  have  common  traits. 

4.  Targets  should  provide  actionable  feedback  to  leadership  and 
organizational  members. 

5.  Performance  measurement  should  integrate  and  unify  the 
organization. 

1.  Strategic  Orientation 

Performance  metrics  should  provide  a  snapshot  of  organizational 
execution  relative  to  strategy;  they  should  reflect  progress  toward  achieving  the 
organization’s  game  plan  (Lynch  and  Cross  1995).  Long-term,  strategic  focus  is 
easily  overlooked  within  federal  agencies,  including  the  Department  of  the 
Defense,  due  to  the  short-term  nature  of  funding  profiles,  to  obligation  and 
expenditure  rates  reviewed  multiple  times  during  the  year,  and  to  the  immediacy 
interjected  by  unplanned  contingencies.  A  reactive  mindset  can  become 
engrained  due  to  short-notice  reporting  requirements  regarding  financial  standing 
and  organizational  accomplishments  (Lynch  and  Cross  1995).  Appropriate 
metrics  should  help  maintain  a  strategic  focus  in  the  midst  of  a  volatile  operating 
environment. 

2.  Critical  Performance  Variables 

In  designing  performance  metrics,  organizations  must  focus  on  what  is 
imperative  to  their  strategic  success,  to  the  exclusion  of  things  that  are  not 
important.  These  important  elements  are  known  as  critical  performance 
variables  (CPV).  Getting  the  CPVs  right  is  arguably  the  most  important  part  of 
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determining  useable  performance  metrics.  The  adage,  “What  gets  measured 
gets  managed,”  applies.  If  what  is  truly  important  to  strategic  success  is  not 
measured,  and  less  important  factors  are,  then  the  result  will  be  the  diversion  of 
management  attention  toward  the  less  significant  factors.  One  way  to  determine 
CPVs  is  to  visualize  where  the  organization  wants  to  be  five  years  hence,  and  if 
at  that  time  the  organization’s  strategy  had  failed,  what  factors  would  have  led  to 
that  failure.  The  factors  that  were  important  enough  to  have  caused  the  strategy 
to  fail  are  CPVs  (Simons  2000). 

Performance  metrics  should  have  a  long-term  perspective  and  be 
indicative  of  how  the  organization  provides  value  to  its  customers.  They  should 
be  balanced  and  focused  on  more  than  just  products  and  financial  outcomes  by 
including  non-financial  measures  (Lynch  and  Cross  1995,  Kaplan  and  Norton 
1996,  Simons  2000).  If  all  aspects  of  the  strategy  are  not  measured,  only  those 
parts  that  are  measured  are  likely  to  get  significant  management  attention. 
Balance  is  particularly  important  for  knowledge-based  organizations  like  PEO 
IWS  because  intellectual  capital  is  not  captured  well  in  financial  metrics.  Kaplan 
and  Norton  identify  four  perspectives  for  balance: 

•  Financial  Perspective  -  the  economic  impact  of  decisions 

•  Customer  Perspective  -  factors  that  are  critical  to  customers 
remaining  loyal  and  satisfied 

•  Business-Internal-Processes  Perspective  -  internal  processes  that 
are  critical  to  the  organization  achieving  its  strategic  objectives 

•  Learning  and  Growth  Perspective  -  identifies  the  infrastructure 
(people,  systems,  and  procedures)  that  will  enable  long-term 
growth  and  continuous  improvement 

Performance  metrics  should  not  only  reflect  the  CPVs;  they  should  be 
causally  linked  and  mutually  reinforcing  (Kaplan  and  Norton  1996).  The  CPVs 
can  be  thought  of  as  the  organization’s  story  of  success.  Kaplan  and  Norton  sum 
it  up  by  emphasizing  the  importance  of  focusing  on  drivers  of  future  performance: 
the  cause-and-effect  variables,  which  link  actions  to  outcomes.  Performance 
metrics  can  measure  CPVs  or  the  desired  outcomes  themselves.  This  will  result 
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in  either  leading  or  lagging  indicators  of  success,  respectively.  A  mix  of  the  two 
is  desirable  (Kaplan  and  Norton  1996).  Simons  further  commented  on  factors 
involved  in  choosing  what  to  measure. 

•  Measure  inputs  when  it  is  impossible  to  measure  processes  or 
outputs,  when  the  cost  of  inputs  is  high  relative  to  outputs,  and 
when  quality  and  safety  is  important. 

•  Measure  processes  when  they  are  observable  and  measurable,  the 
cost  of  measuring  them  is  low,  standardization  is  critical,  cause  and 
effect  relationships  are  understood,  and  it  can  result  in  strategic 
advantage. 

•  Measure  outputs  when  they  are  observable  and  measurable,  cost 
of  measuring  them  is  low,  cause  and  effect  relationships  are  not 
well  understood,  and  freedom  to  innovate  is  desired. 

Choosing  the  right  thing  to  measure  can  save  time,  effort  and  money. 

3.  Desired  Characteristics  of  Performance  Metrics 

Regardless  of  what  is  measured,  the  metrics  should  share  some  common 
characteristics.  They  should  be  actionable,  measuring  items  that  are  within  the 
manager’s  ability  to  influence.  The  information  provided  by  metrics  should  be 
accurate  and  unambiguous.  Metrics  should  capture  the  essence  of  a  CPV. 
Simon’s  summed  up  the  attributes  of  a  good  measure;  it  should  be  objective, 
complete,  and  responsive. 

•  Objective  -  quantifiable  and  measurable  with  confidence  in 
accuracy. 

•  Complete  -  capture  all  relevant  data  in  a  manner  that  is 
understandable  to  users. 

•  Responsive  -  actionable  by  managers  and  leaders;  within  their 
sphere  of  influence. 

Another  way  to  think  of  the  attributes  of  a  good  performance  metric  is  with 
the  acronym  SMART  (Harbour  1997). 

•  S  -  specific 

•  M  -  measurable 

•  A  -  actionable 

•  R  -  relevant 

•  T  -  timely 
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This  list  introduces  the  concept  of  time.  Timeliness  can  be  critical  to  the  impact 
of  a  manager’s  actions.  If  a  situation  requires  immediate  attention,  but  the 
manager  does  not  become  aware  of  the  variance  from  the  performance  standard 
until  30  days  later,  that  is  problematic.  Those  are  30  days  of  potentially  wasted 
organizational  effort  moving  in  an  unintended  direction. 

4.  Tool  for  Management  Action 

Performance  metrics  are  nothing  if  not  a  mechanism  for  management 
action.  To  function  in  this  manner,  metrics  must  be  compared  to  some 
benchmark  to  indicate  progress.  As  indicated  by  the  cybernetic  feedback  model, 
the  metric  is  compared  against  the  standard  to  provide  feedback  to  improve 
inputs  or  processes.  Performance  measurement  provides  a  systematic  process 
for  tracking  progress  and  making  midcourse  corrections;  a  regular  opportunity  for 
strategic  review  (Lynch  and  Cross  1995,  Kaplan  and  Norton  1996).  Because 
metrics  focus  on  things  management  can  influence,  they  direct  attention  toward 
what  can  be  improved  and  away  from  what  is  beyond  the  manager’s  control. 
Feedback,  particularly  when  communicated  throughout  an  organization,  provides 
an  excellent  chance  for  organizational  learning  and  improvement.  In  that  way,  a 
well-designed  measurement  and  feedback  system  can  be  the  catalyst  for 
continuous  improvement  and  strategic  development. 

Appropriate  performance  metrics  should  help  an  organization  achieve  a 
higher  return  on  management  (ROM)  (Simons  2000).  ROM  is  equivalent  to  the 
amount  of  organizational  energy  released  divided  by  the  amount  of  management 
time  expended.  The  goal  of  good  performance  measurement  would  be  to  allow 
leadership  to  manage  by  exception  when  practical,  so  that  they  can  focus  energy 
on  issues  and  processes  that  need  their  continual  expert  attention.  By  allowing 
business  processes  to  operate  without  continual  management  attention,  the 
denominator  of  the  ROM  ratio  is  minimized,  thus  increasing  the  value  of  ROM. 

5.  Performance  Metrics  as  Organizational  Unifiers 

By  integrating  metrics  derived  from  strategy,  the  performance 
measurement  system  can  effectively  communicate  the  chosen  strategy 
throughout  the  organization,  align  and  link  individual,  department  and  corporate 
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goals,  and  identify  and  align  strategic  initiatives  (Lynch  and  Cross  1995,  Kaplan 
and  Norton  1996).  The  right  metrics  should  let  employees  see  how  their  actions 
contribute  to  the  performance  of  the  whole  organization  (Lynch  and  Cross  1995). 
Individual  employee  connection  to  the  entire  organization’s  performance  should 
act  as  a  motivator,  particularly  when  overall  performance  benefits  individuals, 
uniting  the  organization  around  strategic  goals.  The  performance  measurement 
system  serves  to  coordinate  resources  throughout  the  organization  in  line  with 
corporate  vision,  potentially  reducing  redundancy  and  improving  effectiveness 
and  efficiency.  When  coupled  with  a  good  internal  communication  system,  it  also 
provides  an  opportunity  for  bottom-up  strategic  feedback  from  the  elements  of 
the  organization  that  are  frequently  in  closest  contact  with  the  customers. 
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III.  COLLECTED  DATA 


A.  STRATEGIC  OBJECTIVES 

One  of  the  primary  factors  that  might  influence  the  success  of  a 
performance  measurement  system  is  how  well  metrics  are  linked  to  strategy.  In 
order  to  arrive  at  performance  metrics  for  IWS  1.0  and  2.0,  the  strategic 
environment  in  which  they  operate  must  be  understood.  PEO  IWS  reports 
directly  to  the  Assistant  Secretary  of  the  Navy  (Research,  Development,  and 
Acquisition)  (ASN  (R,D&A))  in  acquisition  matters,  and  to  the  Commander,  Naval 
Sea  Systems  Command  (NAVSEA)  for  matters  pertaining  to  in  service  support 
(Department  of  the  Navy  1995).  NAVSEA  subsequently  reports  to  ASN  (RD&A) 
and  the  Chief  of  Naval  Operations  (CNO).  ASN  (RD&A)  and  the  CNO  report  to 
the  Secretary  of  the  Navy. 

1 .  The  Secretary  of  the  Navy 

The  Secretary’s  strategic  goals  were  spelled  out  in  the  Department  of  the 
Navy’s  FY  2006/2007  budget  submission  to  the  Secretary  of  Defense.  They 
include  maintaining  a  ready  and  sustainable  force  to  meet  today’s  challenge, 
investing  in  tomorrow’s  capabilities,  and  establishing  processes  and 
organizations  that  make  effective  and  efficient  use  of  our  scarce  resources. 
These  goals  are  supported  through  four  pillars  of  effective  military  power. 

•  People 

•  Combat  Capability 

•  Technology  Insertion 

•  Improved  Business  Practices 

2.  Chief  of  Naval  Operations 

The  CNO  has  five  stated  strategic  objectives  in  his  2005  guidance  to  the 

Fleet. 

•  Win  the  Global  War  on  Terror  (GWOT). 

•  Improve  readiness  for  global  response. 

•  Integrate  Sea  Strike,  Shield  Basing  and  FORCEnet  into  the  Joint 
Force. 
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•  Capture  funds  through  Sea  Enterprise  to  recapitalize  the  Navy. 

•  Develop  the  21 st  century  workforce. 

He  also  has  five  long  standing  strategic  priorities  which  have  shaped  the  Navy 
during  his  tenure. 

•  Current  readiness 

•  Future  readiness 

•  Alignment 

•  Quality  of  service 

•  Manpower. 

3.  Assistant  Secretary  of  the  Navy  (Research,  Development  and 
Acquisition)  (ASN  (RD&A)) 

The  ASN  (RD&A)  guidance  for  2004  included  the  following  strategic 
objectives  (PEO  IWS  2005). 

•  Improve  warfighter  satisfaction 

•  Shape  and  train  an  effective  acquisition  workforce 

•  Lower  total  ownership  cost  of  equipment  and  services 

•  Reduce  cost  and  cycle  time 

•  Accept  responsibility  as  stewards  of  the  national  interest 

•  Personally  change  and  improve  the  neighborhood  every  day! 

•  Consider  all  solutions  -  high  tech  to  simple,  commercial-off-the- 
shelf  to  military,  U.  S.  to  international 

4.  Naval  Sea  Systems  Command 

The  NAVSEA  guidance  for  2005  contained  the  following  objectives. 

•  Human  capital  strategy  -  continue  to  refine  and  implement 

•  Lean  Implementation  Plan  -  strive  for  efficiency  while  improving 
effectiveness 

•  Capture  Cost  Reduction  -  capture  cost  savings  from  increased 
efficiency  to  return  to  corporate  Navy  and  customer  accounts 

•  Improve  and  Measure  Productivity 

•  Virtual  SYSCOM 

5.  PEO  IWS 

PEO  IWS  identified  areas  for  strategic  success  during  strategic  planning 
meetings  at  the  beginning  of  FY  2005  (PEO  IWS  2005). 
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•  Personnel  -  optimum  staffing,  quality  of  work  force,  quality  of  life  for 
employees,  and  refine  human  capital  and  horizontal  integration 
strategies 

•  Programmatic  -  accurately  manage  cost,  schedule, 
technical/performance,  and  risk  and  achieve  success  in 
development  of  Open  Architecture,  CGX  Radar,  and  Active  Missile 

•  Business  -  proactively  manage  all  aspects  of  the  budget  process 

•  Fleet  Readiness  -  support  the  ultimate  customer 

Figures  3  and  4  depict  the  goals  and  perspectives  listed  above  in 
alignment  from  SECNAV  down  through  PEO  IWS.  PEO  IWS  is  depicted  in  its 
two  separate  chains  of  commands.  If  performance  metrics  within  PEO  IWS  1.0 
and  2.0  are  to  contribute  to  organizational  success  and  mission  accomplishment, 
it  is  important  that  they  support  PEO  IWS  strategy  and  the  other  strategies  in 
their  chain  of  command.  In  this  way  the  actions  at  the  major  program  level  within 
PEO  IWS  should  impact  the  entire  enterprise  strategy  in  a  positive  manner.  It  is 
apparent  that  the  different  strategic  objectives  are  aligned  and  support  one 
another  in  achieving  the  ultimate  goal  of  effective  military  power.  The  lines 
connecting  the  various  goals  and  perspectives  are  the  interpretation  of  the 
author. 
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Figure  3.  Strategic  Alignment  of  PEO  IWS  Perspectives 
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Figure  4.  Strategic  Alignment  of  PEO  IWS  Perspectives 
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B.  PEO  IWS  1 .0  AND  2.0  MISSION  AND  FUNCTION 

The  primary  mission  for  PEO  IWS  is  to  provide  the  United  States  Navy 
technically  superior  warfighting  capability  to  take  the  fight  to  the  enemy  and  win. 
Within  PEO  IWS,  IWS  1.0  and  2.0  contribute  to  the  accomplishment  of  that 
mission  by  providing  top-quality  integrated  combat  systems  and  above  water 
sensors,  respectively.  Neither  IWS  1.0  nor  2.0  physically  produce  weapon 
systems.  Instead,  they  contract  both  Navy  engineering  field  activities  and  private 
defense  contractors  to  produce  those  systems.  IWS  1.0  and  2.0  provide  “cradle 
to  grave”  service  to  the  warfighter  from  concept  refinement  through  system 
disposal  to  ensure  the  Fleet  has  the  needed  weapons  systems  to  execute  their 
missions. 

With  that  in  mind,  three  middle  to  upper  level  leaders/managers  at  both 
IWS  1.0  and  2.0  were  interviewed  with  the  goal  of  defining  what  success  looked 
like  for  their  organizations,  and  also  determining  what  factors  influenced 
achieving  that  vision  of  success.  An  attempt  was  made  to  avoid  leading 
questions  in  an  effort  to  let  the  interviews  reflect  the  feelings  of  the  interview 
subjects  alone.  From  those  initial  interviews,  causal  performance  maps  were 
constructed  for  each  interview.  Those  maps  were  then  presented  to  the 
interview  subjects  to  modify  as  they  felt  necessary  to  best  tell  their  story  of  how 
to  achieve  organizational  success.  Through  this  process  of  interview  and 
subsequent  refinement,  causal  performance  maps  became  more  accurate  from 
the  perspective  of  individual  interview  subjects.  The  interview  subjects  were  not 
asked  to  critique  each  others’  inputs,  only  to  concentrate  on  their  own 
perspectives  as  to  what  were  the  drivers  of  success  for  their  major  program  area. 

Subsequent  to  the  editing  of  individual  maps,  one  composite  map  was 

developed  for  IWS  1.0  and  one  for  IWS  2.0.  This  was  done  by  combining  the 

separate  maps  within  each  major  program  area.  Where  differences  existed 

between  maps,  an  attempt  was  made  to  resolve  the  difference  through  further 

communication  with  the  interview  subjects.  If  consensus  was  not  feasible  then 

both  versions  were  included  in  the  composite  map.  Since  these  maps  are  to  be 

used  to  derive  recommended  useful  performance  metrics,  it  was  important  to 
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capture  multiple  points  of  view  to  get  as  whole  a  picture  as  practicable.  Some 
consolidation  of  similar  variables  was  required  to  make  the  maps  less  cluttered 
and  more  comprehensible,  but  effort  was  made  to  maintain  the  intent  of  the 
original  subjects’  maps. 

C.  INTERVIEWS 

Six  subjects  within  IWS  1 .0  and  2.0,  three  from  each  major  program,  were 
selected  to  be  interviewed  based  on  their  broad  acquisition  experience.  The  six 
subjects  had  a  total  of  112  years  of  either  military  or  acquisition  experience. 
Additionally,  all  were  in  positions  within  IWS  1.0  and  IWS  2.0  to  have  detailed 
knowledge  of  both  strategic  direction  and  day-to-day  operations.  Five  of  the 
subjects  were  Government  Service  employees  and  one  was  a  Naval  Officer. 
Due  to  demands  of  the  individual  subjects’  work  and  travel  schedules,  and 
limitations  in  the  author’s  availability  to  travel,  two  of  the  subjects  were  initially 
interviewed  together.  Those  two  were  subsequently  asked  to  critique  the 
resultant  causal  performance  map  independently. 

The  subjects  were  interviewed  using  the  protocol  depicted  in  Table  1. 
They  were  asked  to  define  what  desired  outcomes  translated  into  organizational 
success  for  IWS  1 .0  or  2.0,  respectively.  The  interview  subjects  were  next  asked 
to  lead  the  author  through  the  chain  of  cause  and  effect  processes  and  inputs 
that  would  result  in  the  previously  identified  desired  outcomes.  The  subjects’ 
responses  were  then  analyzed  to  identify  the  significant  variables  that  would 
influence  the  desired  outcomes.  Those  variables  and  how  they  interact  with  one 
another  were  then  depicted  in  the  causal  performance  maps. 
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1 .  Record  time,  place,  and  identity  of  interviewee. 

2.  Indicate  that  the  interviewee  will  remain  anonymous  and  record  consent  to 
being  recorded. 

3.  Could  you  describe  your  current  position  and  work  at  IWS? 

4.  What  is  the  mission  of  IWS  2.0  (or  1 .0,  as  appropriate)? 

5.  Does  your  unit  have  overall  goals? 

6.  Please  describe  what  would  be  considered  a  good  performance  outcome  for 
your  unit? 

7.  If  you  could  tell  a  story  about  how  your  unit  can  succeed,  what  would  that 
story  be  like?  Can  you  tell  that  story? 

8.  What  factors  are  most  important  in  determining  whether  your  unit  meets  its 
goals,  achieves  good  performance,  or  meets  performance  targets?  Possible 
follow-up  examples: 

A.  People  investments  (training,  education,  etc) 

B.  Financial  efficiency  (Waste) 

C.  Technical  performance  of  new  systems  (Quality) 

D.  Supportability  of  new  system 

E.  Delivery  of  new  systems  (Delivery) 

F.  Time  efficiency,  time  to  complete  tasks  (Cycle  time) 

9.  Are  these  factors  linked  in  any  way? 

10.  Can  these  factors  be  measured? 

1 1 .  Would  any  of  these  measure  help  you  manage  your  organization  any  better 
to  achieve  success? 

12.  Now  look  out  five  years,  are  any  other  factors  critical  to  IWS’  success? 

13.  Who  are  your  unit’s  customers,  internal  and  external? 

14.  Do  you  get  feedback  on  their  satisfaction? 

15.  What  is  the  biggest  indicator  of  failure  for  IWS? 


Table  1.  Initial  Interview  Protocol 
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1.  Causal  Performance  Maps 

The  maps  that  resulted  from  the  interviews  with  IWS  1.0  and  2.0  subjects 
are  depicted  in  Figures  5  through  1 1 .  The  interview  subjects  are  identified  as 
Subjects  A  through  F  for  the  purpose  of  anonymity.  Outcomes,  the  definition  of 
success,  appear  on  the  right  side  of  each  diagram  with  factors  that  lead  to  that 
outcome  appearing  to  the  left.  Arrows  show  the  causal  links  between  inputs, 
processes,  and  outcomes.  Double-headed  arrows  indicate  that  factors  influence 
each  other.  Terminology  has  been  standardized  to  the  maximum  extent  without 
disturbing  the  intent  of  the  interview  subjects. 

Figure  7  is  a  composite  map  for  IWS  1.0  derived  from  combining  the 
causal  performance  maps  of  the  different  IWS  1.0  subjects.  Figure  11  is  the 
composite  map  for  IWS  2.0.  The  maps  can  be  difficult  to  interpret,  largely  due  to 
the  many  lines  indicating  causal  relationships.  One  of  the  main  take-aways, 
which  are  relatively  easier  to  identify,  are  the  CPVs  themselves.  As  stated 
earlier,  getting  the  CPVs  right  can  be  one  of  the  most  difficult  parts  of 
determining  performance  metrics.  It  should  be  noted  that  some  variables  from 
the  individual  maps  have  been  combined  to  enhance  the  readability  of  the 
composite  maps. 
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Figure  6.  Causal  Performance  Map  -  IWS  1.0  Subjects  B  and  C 
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Figure  7.  Causal  Performance  Map  -  IWS  1.0  Composite 
a.  IWS  1.0 

(1)  Outcomes.  Figures  5  through  7  are  relevant  to  IWS 
1.0,  with  Figure  6  being  the  map  produced  for  two  interview  subjects  who  were 
interviewed  together.  The  outcomes  that  defined  success  were  slightly  different 
for  the  two  maps,  but  both  included  the  military  acquisition  standards  of  cost, 
schedule,  and  technical  performance.  Of  note,  supportability  of  deployed 
systems  was  viewed  as  a  subset  of  technical  performance  within  IWS  1.0. 
Additionally,  they  indicated  it  was  critical  to  maintain  the  funding  stream  for 
current  programs.  If  the  major  program  manager,  IWS  1.0,  could  end  a 
hypothetical  period  with  all  programs  on  schedule,  on  budget,  meeting  technical/ 
performance  requirements,  and  have  a  secure  source  of  funding  that  would  be  a 
successful  outcome  from  the  perspective  of  those  interviewed. 

Subject  A  also  included  the  improvement  of  the  IWS  1.0 
team  as  a  successful  outcome.  The  subject  alluded  to  the  importance  of  the 
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professional  ability  of  people  within  the  organization.  There  was  also  a  concern 
about  the  future  retirement  of  many  PEO  IWS  employees  and  a  need  to  maintain 
a  continuous  recruitment  and  development  program  within  the  organization  to 
alleviate  any  loss  in  organizational  capability  from  retirements. 

Subject  C  added  an  important  topic,  customer  satisfaction. 
His  comments  mostly  concerned  Fleet  users  of  IWS  1.0’s  ultimate  product, 
integrated  combat  systems.  Customer  satisfaction  is  recognized  in  all  the 
previously  mentioned  literature  as  the  key  to  survivability  and  growth  in  private 
sector  organizations  (Lynch  and  Cross  1995,  Kaplan  and  Norton  1996,  Simons 
2000).  The  fact  that  subject  C  mentioned  it  recognized  that  it  is  important  to  the 
perceived  success  of  IWS  1.0,  as  well.  The  fact  that  it  was  not  mentioned 
specifically  by  subjects  A  or  B  does  not  mean  that  it  is  insignificant,  only  that  it 
was  not  specifically  mentioned. 

(2)  Processes.  Significant  processes  brought  out  by  the 
interview  subjects  included:  communication,  teamwork,  weapon  system  design, 
contract  design,  contractor  performance,  and  financial  management.  All 
recognized  the  importance  of  human  capital,  as  well,  but  subject  A  gave  it  much 
more  emphasis  and  included  it  not  only  as  an  input,  but  also  within  processes. 
Additionally,  subjects  B  and  C  gave  more  emphasis  to  program  planning  and  test 
and  evaluation’s  (T&E)  impact  on  ultimate  outcomes  of  the  organization 

Communication  is  a  key  process  that  ultimately  impacts  all  of 
IWS  1.0’s  outcomes.  As  interview  subject  C  put  it,  it  is  a  basic  requirement  that 
underlies  all  planning  and  execution  within  the  organization.  It  includes 
communication  within  IWS  1.0,  between  IWS  1.0  and  the  rest  of  PEO  IWS,  and 
communication  with  all  stakeholders  outside  PEO  IWS. 

Teamwork  is  closely  linked  with  communication.  It  concerns 
internal  teamwork  within  the  separate  facets  of  IWS  1 .0,  external  teamwork 
within  the  rest  of  PEO  IWS,  and  external  teamwork  with  outside  stakeholders. 
Teamwork  also  encompasses  cooperation  between  separate  private  contractors 
which  can  be  problematic  because  of  their  concerns  about  proprietary  knowledge 
and  maintaining  competitive  advantage.  IWS  1 .0  is  the  lead  major  program  for 
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integrating  the  separate  parts  within  PEO  IWS  by  virtue  of  its  mission  to  provide 
integrated  combat  systems  to  the  fleet.  By  the  nature  of  its  work  it  must  link  the 
efforts  of  the  other  major  programs  within  PEO  IWS  into  a  useful,  total  system  for 
the  warfighter.  Integration  of  separate  processes  is  at  the  forefront  of  military 
acquisition  as  noted  by  the  emphasis  on  integrated  product  and  process 
development  (IPPD)  in  the  Defense  Acquisition  Handbook..  IPPD  emphasizes  a 
multidisciplinary  approach  to  problem  solving  and  meeting  customer  and 
stakeholder  needs. 

Human  capital  development  is  an  important  aspect  of 
organizational  growth  which  has  impacts  throughout  IWS  1.0’s  processes  and 
outcomes.  Not  only  does  it  impact  the  obvious  capability  of  employees,  but  also 
feeds  back  to  recruitment  and  retention.  Current  and  future  employees  are  likely 
to  be  motivated  to  work  for  an  organization  that  allows  them  to  improve  their 
abilities  and  skill  sets,  regardless  of  the  organization’s  motive  in  offering  that 
opportunity. 

Financial  management  recognizes  that  IWS  1.0  needs  to 
intelligently  manage  its  funding  to  deliver  on  its  mission;  sustainable  funding  is  a 
key  enabler  of  success.  The  negative  consequences  from  poor  financial 
management  are  cost  overruns,  schedule  slippage,  and  rescission  of  funds  by 
higher  authority. 

Contract  design  implies  that  the  type  of  contract  is  important 
to  contractor  performance  and  T&E  success.  Subject  A  emphasized  the  need  to 
incentivize  the  provider  of  service  to  meet  cost,  schedule  and 
technical/performance  milestones,  as  well  as  to  be  integrated  with  other 
contractors.  The  first  three  of  those  topics  are  already  well  incorporated  within 
the  acquisition  communities  contracting  process.  They  are  largely  driven  by 
technological  maturity  and  by  a  determination  of  which  party  to  the  contract  will 
bear  the  risks  of  overruns  and  performance  failure.  Regarding  degree  of 
integration,  subject  A  emphasized  that  contractors  could  be  financially  motivated 
to  interact  well  with  other  defense  contractors  working  on  a  program  to  ensure 
the  program  met  all  of  its  goals.  Specifically,  at  least  part  of  any  incentive  bonus 
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would  be  paid  only  if  all  involved  contractors  met  their  goals.  Thus,  contractors 
would  be  encouraged  to  help  one  another  and  work  together  in  a  truly  integrated 
program. 

Weapon  system  design  is  influenced  by  the  degree  of 
teamwork,  communication,  skill  of  the  personnel  involved,  and  specificity  of  the 
stated  requirements.  As  stated  earlier,  the  IPPD  process  is  significant  and 
emphasizes  knowledgeable  personnel  from  multiple  disciplines  working  together 
in  an  open  manner  to  develop  the  best  design  within  stated  parameters. 
Subjects  B  and  C  included  it  within  the  more  encompassing  heading  of  program 
planning. 

Contractor  performance,  according  to  interview  subjects,  is 
not  fully  within  the  major  program  manager’s  direct  control,  but  is  certainly  within 
his  or  her  influence.  In  a  large  sense,  one  of  the  primary  responsibilities  of 
acquisition  organizations  in  the  military  is  to  influence  contractor  performance. 
Much  of  that  influence  is  brought  about  by  the  previously  mentioned  processes: 
communication,  teamwork,  specificity  of  the  contract,  and  a  good  system  design. 

Subjects  B  and  C  indicated  that  program  planning  plays  a 
key  role  in  the  ultimate  outcomes  of  the  organization.  That  planning  included,  but 
was  not  limited  to,  acquisition  strategy,  maintenance  strategy,  including  an 
integrated  logistics  support  plan,  a  development  strategy,  and  system  design. 
Program  planning  has  a  significant  subsequent  impact  on  contract  design  which 
further  influences  contractor  performance  and  ultimately  organizational 
outcomes. 

Test  and  evaluation  can  have  a  major  impact  on  ultimate 
technical/performance,  cost  and  schedule.  The  subjects  indicated  that  the  best 
way  to  ensure  success  was  through  robust  modeling  and  simulation  with 
sequentially  increasing  levels  of  fidelity  to  ensure  the  system  is  ready  to  perform 
before  field  tests  are  attempted.  Making  sure  that  preliminary  testing  is  adequate 
saves  money  and  time  in  the  long  run  by  not  having  to  repeat  more  expensive 
operational  and  live  fire  tests  because  of  system  failure. 
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Risk  management  was  depicted  by  subjects  B  and  C  with 
double-headed  arrows  between  cost,  schedule,  and  technical/performance.  Risk 
can  be  defined  as  the  potential  inability  to  achieve  overall  program  objectives 
within  defined  cost,  schedule,  and  technical  constraints  (Naval  Postgraduate 
School  2004)  If  there  is  no  slack  in  either  budget  or  schedule,  and  a  program  is 
under-performing  technically,  the  possible  outcomes  that  might  achieve  technical 
goals  are  an  increase  in  cost  or  a  delay  in  schedule.  Risk  is  continually 
assessed  by  the  program  manager  to  determine  status  of  the  program  and 
forecast  probability  of  future  success. 

(3)  Inputs.  While  all  IWS  1.0  subjects  recognized  human 
capital  as  a  basic  input  for  their  processes,  subjects  B  and  C  recognized  that  the 
degree  of  technological  maturity,  the  specificity  of  the  system  requirements,  and 
the  initial  funding  for  any  program  were  all  important  ingredients  to  ultimate 
success. 

It  can  generally  be  said  that  any  organization  will  function 
better  if  managed  and  operated  by  highly  qualified  and  trained  personnel  as 
opposed  to  lesser  skilled  manpower.  IWS  1 .0  does  not  have  significant  levels  of 
facilities  and  infrastructure;  all  the  subjects  recognized  that  the  primary  capital  for 
IWS  1.0  resides  in  its  people.  Hiring,  motivating,  and  retaining  their  human 
capital  will  be  critical  to  the  long-term  success  of  the  organization. 

Mature  and  specific  weapon  system  requirements  are 
brought  about  by  thorough,  integrated  analysis  and  they  tend  to  eliminate 
ambiguity  for  the  manufacturer.  Mature  technology  removes  risk  associated  with 
developing  unproven  or  experimental  technology. 
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Figure  8.  Causal  Performance  Map  -  IWS  2.0  Subject  D 
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Figure  9.  Causal  Performance  Map  -  IWS  2.0  Subject  E 
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Figure  10.  Causal  Performance  Map  -  IWS  2.0  Subject  F 
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Figure  1 1 .  Causal  Performance  Map  -  IWS  2.0  Composite 
b.  IWS  2.0 

Figures  8  through  1 1  were  derived  from  the  interviews  within  IWS 
2.0,  with  Figure  1 1  being  a  composite  map. 

(1)  Outcomes.  As  in  IWS  1.0  technical/performance, 
schedule  and  cost  were  all  identified  as  common  outcomes.  Subjects  D  and  E 
also  separated  out  supportability  as  a  key  outcome.  Subjects  E  and  F  identified 
the  talent,  growth  and  optimal  deployment  of  the  IWS  2.0  workforce.  Subject  E 
was  very  focused  on  customer  satisfaction,  as  well. 

Subject  F  brought  out  the  importance  of  electronic  warfare 
(EW)  and  radar  operating  in  unison  as  an  integrated  system.  The  thinking  is  that 
if  EW  systems  can  be  well  integrated  into  an  air  defense  combat  systems 
solution  for  the  warfighter,  the  total  system  should  have  equal  or  greater  combat 
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effectiveness  at  a  lower  total  cost.  Much  of  the  hypothetical  savings  would  come 
from  a  need  to  procure  and  expend  fewer  surface-to-air  missiles. 

Subject  F  also  identified  the  importance  of  new  project 
creation,  where  it  made  sense  to  the  Navy  as  a  whole.  Because  of  their  work 
with  the  Fleet  and  industry,  IWS  2.0  personnel  are  exposed  to  both  the  needs  of 
Navy  operators  and  emerging  technologies  within  industry.  When  they  can 
marry  the  two  efficiently  and  effectively  through  a  new  project  it  can  be  the  best 
desired  outcome  for  all,  including  tax  payers. 

(2)  Processes.  Commonly  recognized  factors  by  all  three 
IWS  2.0  subjects  were  communications,  teamwork,  contract  design,  program 
planning,  T&E,  and  contractor  performance.  Subject  D  identified  financial 
management,  thorough  technical  and  program  reviews  and  JCIDS  requirements 
as  additional  determinants  of  organizational  outcomes.  Subject  F  acknowledged 
the  importance  of  regular  and  honest  assessment,  very  similar  to  the  review 
process  mentioned  by  subject  D.  Subject  F  also  highlighted  the  risk 
management  tradeoffs  between  technical/performance,  cost  and  schedule,  as 
well  as  the  risk  mitigation  that  can  be  derived  from  international  cooperation. 
Lastly,  subject  F  highlighted  the  process  of  comparing  current  system  capability 
against  available  options  for  improvements  to  arrive  at  a  more  capable  and/or 
cost  effective  weapon  system  solution.  The  IWS  2.0  subjects  shared  many  of  the 
same  processes  with  their  IWS  1 .0  counterparts,  and  to  avoid  redundancy,  this 
commentary  will  be  limited  to  any  differences  from  what  was  discussed  for  IWS 
1.0. 


Teamwork  took  on  a  less  prominent  role  in  the  IWS  2.0 
organization  largely  because  the  weapon  systems  that  they  work  with  do  not 
combine  inputs  from  other  major  programs  within  PEO  IWS  to  the  same  extent 
that  they  do  in  IWS  1 .0.  That  is  not  to  say  that  degree  of  integration  was 
unimportant,  but  it  was  not  as  significant  as  mentioned  for  IWS  1 .0. 

Contract  design  was  an  important  aspect  of  IWS  2.0 
processes  and  an  additional  factor  was  mentioned,  the  need  not  only  for 
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specificity,  but  for  the  elements  of  the  contract  to  be  measurable.  In  order  to  hold 
parties  to  the  contract  accountable,  and  help  determine  contract  awards,  more 
measurable  specific  attributes  are  better. 

Risk  management  and  its  importance  to  organizational 
outcomes  was  emphasized  by  subject  F.  Part  of  honestly  assessing  T&E 
progress  and  current  system  standing  is  comparison  to  the  required  scope 
covered  in  the  requirements  document.  If  progress  is  not  adequate,  either  due  to 
cost,  performance,  or  schedule  considerations,  trade-offs  may  be  made  to  keep 
the  product  either  on  time,  on  budget  or  aimed  at  accomplishing 
technical/performance  targets. 

The  achievement  of  EW  and  radar  integration  would  be 
influenced  by  acceptance  of  an  EW  system’s  ability  to  detect  and  deceive 
perceived  threats  to  the  ships  employing  the  system.  That  acceptance  would 
hinge  greatly  on  the  credibility  of  test  results,  to  the  extent  that  the  ultimate  user 
could  trust  and  have  confidence  in  them. 

International  cooperation  in  weapon  system  design  allows 
IWS  2.0  to  leverage  mature  technologies  and  experience  from  other  nations  to 
create  better  or  less  risky  solutions  to  solve  fleet  problems.  That  cooperation  can 
only  take  place  in  an  environment  of  trust  where  ongoing  dialogue  exists.  The 
benefits  of  this  interaction  come  from  the  different  perspectives  that  different 
cultures  might  bring  to  the  same  problem.  New  methods  of  solving  perplexing 
problems  have  been  gained  in  the  past  from  altering  the  problem  solver’s 
paradigm. 

New  project  creation  is  influenced  by  the  aforementioned 
international  cooperation,  but  also  by  keeping  a  keen  eye  on  emerging 
technologies  and  assessing  them  against  current  systems.  Existing  systems  can 
become  obsolete  when  the  technology  they  employ  is  no  longer  supported  by 
industry.  In  that  case,  it  would  help  the  Fleet  by  ultimately  improving 
effectiveness  and  system  reliability  for  IWS  2.0  to  nominate  new  technology 
solutions  to  the  resource  sponsor. 
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(3)  Inputs.  All  three  subjects  agreed  that  human  capital  and 
mature  system  requirements  were  essential  inputs  that  ultimately  led  to  IWS 
2.0’s  organizational  outcomes.  Also,  subjects  D  and  F  included  contractor 
experience  and  past  performance  while  subjects  E  and  F  mentioned 
technological  maturity  of  the  system  as  influential  inputs. 

Contractor  experience  and  past  performance  recognize  that 
knowledge  of  the  military  acquisition  process  matters.  Contractors  that  have 
worked  in  the  field  previously  should  be  able  to  better  gauge  what  will  be 
required  to  successfully  produce  for  the  military  again.  Thus,  their  contract  bids 
should  be  more  accurate.  Regarding  past  performance,  it  could  be  a  good 
indicator  of  future  performance. 
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IV.  ANALYSIS  AND  RECOMMENDATIONS 


The  causal  performance  maps  developed  for  IWS  1.0  and  2.0  provide  a 
clear  window  into  the  operations  of  these  two  major  programs  and  illuminate 
variables  that  help  them  achieve  their  desired  outcomes.  Based  on  interview 
subject  input  and  analysis  of  the  maps,  CPVs  were  chosen  that  capture  the 
critical  variables  in  achieving  success.  Those  CPVs  should  answer  the  question, 
“Are  we  doing  the  right  things?”  CPVs  were  analyzed  to  determine  if  they  should 
have  an  input,  process  or  output  metric  based  on  the  criteria  articulated  in 
Chapter  II.  The  chosen  performance  metrics  attempt  to  answer  the  question, 
“Are  we  doing  the  right  things  well?”  CPVs  and  metrics  were  then  organized  into 
a  balanced  scorecard  framework  aligned  with  PEO  IWS  and  enterprise  strategic 
goals. 

One  thing  that  was  evident  from  the  causal  performance  maps  was  that 
IWS  1 .0  and  2.0  have  many  inputs,  processes  and  outputs  in  common.  They  are 
similar  organizations  with  similar  aims.  Table  2  displays  the  inputs,  processes 
and  outcomes  for  both  IWS  1.0  and  2.0  and  then  identifies  those  that  are 
common  to  both.  The  following  analysis  applies  to  both  IWS  1.0  and  2.0  and  is 
based  on  the  list  of  common  CPVs. 

It  was  apparent  that  success  for  both  IWS  1.0  and  2.0  is  influenced  by 
more  than  just  cost,  schedule  and  technical  performance.  To  a  great  extent, 
these  three  outcomes  define  success  for  IWS  1.0  and  2.0,  but  they  are  not  the 
only  outcomes  that  will  help  IWS  1.0  and  2.0  contribute  to  their  long-term  goals. 
They  are  only  lagging  indicators  of  success.  Monitoring  some  of  the  leading 
CPVs  should  provide  leading  indicators  of  success  or  identify  problems  that 
require  a  manager’s  preemptive  action.  Contained  in  the  following  analysis  are 
potential  performance  metrics.  It  is  important  to  reiterate  that  these  are 
recommendations  only.  They  are  derived  from  the  interviews,  past  research  by 
other  authors,  strategic  Navy  documents,  and  the  author’s  experience.  It  is  rare 
that  one  person  can  individually  determine  valuable  metrics  for  an  organization, 
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and  if  a  single  person  could,  they  would  need  to  be  intimately  familiar  with  both 
the  organization  and  its  industry.  Typically,  almost  universally,  good  metrics 
development  requires  integrated  input  and  analysis  from  throughout  an 
organization.  The  following  suggested  metrics  should  be  viewed  as 
recommendations  for  an  organization-wide  metrics  process  at  PEO  IWS,  and  as 
focus  points  for  IWS  1 .0  and  2.0  to  consider  when  evaluating  potential  metrics  for 
their  major  program  areas.  Also,  there  are  more  CPVs  and  metrics  listed  here 
than  IWS  1.0  or  2.0  will  likely  choose  to  implement.  The  following  list  is  drawn 
from  the  maps  and  intended  to  be  a  broad  list  that  the  major  program  managers 
can  choose  from  or  modify  at  their  discretion. 

It  appears  that  to  just  measure  outcomes  would  leave  a  rich  avenue  of 
leadership  influence  without  any  guideposts  to  aid  in  decision  making.  While 
many  good  managers  and  leaders  intuitively  emphasize  all  areas  that  influence 
strategic  success,  not  all  do.  Developing  metrics  for  all  areas  of  the  strategy  will 
ensure  all  parts  of  the  strategy  get  management  attention  (Kaplan  and  Norton 
1996).  Additionally,  it  will  communicate  to  all  levels  of  IWS  1.0  and  2.0  what  is 
viewed  as  important  to  the  major  program  managers.  Providing  a  mix  of  leading 
and  lagging  indicators  will  provide  managers  with  an  ability  to  take  action  before 
outcomes  are  decided  as  well  as  measure  those  ultimate  outcomes. 

The  scorecard  for  IWS  1 .0  and  2.0  displayed  later  in  this  chapter  has  been 
organized  along  the  format  put  forth  by  Kaplan  and  Norton  (1996).  The  four 
classic  perspectives  of  learning  and  growth,  internal  business  processes, 
customer,  and  financial  have  been  replaced  with  the  strategic  perspectives 
identified  by  PEO  IWS.  The  Kaplan  and  Norton  perspectives  are  listed  in 
parenthesis  where  they  match  with  the  more  specific  PEO  IWS  perspectives. 
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IWS  1.0 

IWS  2.0 

Common 

Inputs 

Inputs 

Inputs 

Technology  Maturity 

Technology  Maturity 

Technology  Maturity 

Human  Capital 

Human  Capital  Strategy 

Human  Capital 

Weapon  System 

Weapon  System 

Weapon  System 

Requirements 

Requirements 

Contractor  Experience 

Requirements 

Processes 

Processes 

Processes 

Teamwork 

Teamwork 

Teamwork 

Program  Planning 

Planning 

Planning 

Contract  Design 

Contract  Design 

Contract  Design 

Test  and  Eval 

Test  and  Eval 

Test  and  Eval 

Contractor  Performance 

Contractor  Performance 

Contractor  Performance 

Communication 

Communication 

Communication 

Human  Capital 

Human  Capital 

Human  Capital 

Development 

Development  and 

Development 

Financial  Management 

Assessment 

Financial  Management 

FRP  Surge  Status 

Financial  Management 

Risk 

Ship  Construction 

Review  Process 

Schedule 

Administration 

Risk 

Risk 

International  Cooperation 
Warfighter  Tactical 

Mindset 

Assessment  of  Current 
System  vs.  Future 

Needs 

Innovation 

Outcomes 

Outcomes 

Customer  Satisfaction 

Outcomes 

Customer  Satisfaction 

Schedule 

Customer  Satisfaction 

Schedule 

Cost 

Schedule 

Cost 

Technical/Performance 

Cost 

Technical/Performance 

Capability  of  IWS  Team 

T  echnical/Performance 

Capability  of  IWS  Team 

Secure  Funding 

Talented  Suitably 
Deployed  Workforce 
Supportability 

EW  and  Radar 

Integration 

New  Project  Creation 

(Talented  Workforce) 

Table  2.  IWS  1.0  and  2.0  Common  Critical  Performance  Variables 
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The  scorecard  organizes  the  metrics  under  perspectives  which  show  their 
strategic  alignment.  The  four  perspectives  in  IWS  1 .0  and  2.0’s  scorecard  are 
aligned  with  NAVSEA  and  corporate  Navy  strategic  objectives  as  depicted  in 
Figure  3  and  4.  By  then  aligning  organizational  strategic  objectives,  CPVs,  and 
metrics  under  those  four  perspectives,  the  users  are  able  to  see  how  their  efforts, 
as  measured  through  the  metrics,  can  contribute  to  not  only  IWS  1.0  and  2.0 
success,  but  also  PEO  IWS,  NAVSEA,  and  Navy  strategic  success.  That  ability 
to  tie  disparate  parts  of  the  organization  together,  focus  their  energy  on  strategic 
goals,  and  provide  relevance  to  employee  efforts  is  one  of  the  main  benefits  of  a 
balanced  scorecard  and  performance  measurement  system. 

As  stated  earlier,  the  list  of  metrics  is  longer  than  would  likely  be  chosen  to 
be  monitored  by  IWS  1.0  and  2.0,  but  it  is  intended  to  serve  as  a  list  of  possible 
metrics  to  choose  from  or  modify.  Thus,  the  scorecard  is  spread  across  three 
figures  in  order  to  present  all  the  potential  metrics.  Any  italicized  CPVs  and 
metrics  were  those  that  were  identified  only  by  IWS  2.0  subjects.  The  boldfaced 
metrics  are  those  that  appear  to  be  currently  collected  and  utilized  at  IWS  1.0 
and  2.0.  Just  because  a  metric  is  not  boldfaced  does  not  mean  the  data  are  not 
being  accumulated,  it  merely  means  that  they  may  not  formally  be  reported  to  the 
major  program  managers.  The  analysis  as  to  what  was  currently  being  utilized 
by  IWS  1 .0  and  2.0  was  somewhat  subjective  because  much  of  what  is  currently 
monitored  is  done  so  informally. 


A.  CRITICAL  PERFORMANCE  VARIABLES  AND  USEFUL  METRICS 

The  CPVs  are  inextricably  linked  together  in  a  web  of  causal  links, 
particularly  human  capital,  communication  and  teamwork.  The  causal  maps 
bring  this  out  by  the  web  of  lines  between  CPVs.  The  spider  web  of  links  can  be 
initially  intimidating  to  the  user,  but  persistence  in  evaluating  the  maps  can  bring 
insight  into  factors  that  impact  IWS  1.0  and  2.0.  The  following  analysis  is 
followed  by  a  potential  balanced  scorecard  for  IWS  1 .0  and  2.0  that  lists  possible 
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metrics  for  both  major  programs.  The  following  CPVs  are  arranged  by  balanced 
scorecard  perspectives  with  IWS  2.0  specific  CPVs  discussed  within  the 
Programmatics  perspective. 

1.  Personnel 

a.  Human  Capital 

Human  capital  is  critically  important  to  the  health  and  success  of 
both  IWS  1.0  and  2.0.  “People  are  our  greatest  asset,”  is  a  commonly  used 
adage  in  the  Navy.  In  fact,  at  IWS  1.0,  the  following  statement  is  posted  on  one 
of  the  office  doors,  “It’s  the  PEOPLE,  it’s  the  PEOPLE,  it’s  the  PEOPLE.”  That 
sentiment  is  never  truer  than  in  a  knowledge-based  organization  like  PEO  IWS. 
The  keys  to  organizational  success  are  contained  within  the  people  themselves 
in  the  form  of  their  expertise  and  experience.  This  was  very  apparent  in  the 
maps  for  both  IWS  1.0  and  2.0.  Human  capital  ultimately  affects  all  other 
outcomes. 

Any  leadership  pipeline  has  to  begin  with  attracting  employees, 
some  of  whom  will  become  the  future  leadership  within  the  organization.  Those 
employees  need  to  be  of  sufficient  quality  that  they  can  excel  at  their  current  jobs 
and  learn  new  skills  to  move  into  areas  of  greater  responsibility.  PEO  IWS  has  a 
senior  workforce,  with  over  half  of  its  278  civilian  employees  at  the  GS-14  grade 
or  higher.  It  was  apparent  from  interviewees’  comments  that  many  of  these 
workers  will  be  eligible  for  retirement  during  the  coming  decade.  This  is  a  long¬ 
term  problem  with  a  long-term  solution  and  PEO  IWS  must  address  this  potential 
future  human  capital  loss  now  in  order  to  minimize  loss  in  capability  as  the 
current  workforce  retires.  Largely  due  to  budget  pressures  on  manpower 
accounts,  hiring  is  controlled  at  the  NAVSEA  level  and  is  outside  the  major 
program  manager’s  direct  control.  However,  the  major  program  manager  must 
remain  aware  of  the  current  manning  situation  in  order  to  apply  whatever 
pressure  available  to  the  chain  of  command  to  hire  appropriate  personnel.  It  is 
unclear  if  the  new  National  Security  Personnel  System  will  allow  managers  more 
direct  control  over  hiring  personnel.  One  indicator  of  the  health  of  the  leadership 
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pipeline  might  be  the  number  of  interns  working  within  IWS  1 .0  and  2.0.  A  large 
number  of  interns  indicate  that  there  are  fresh,  talented  newcomers  to  replace 
retiring  employees. 

One  area  of  the  human  capital  equation  that  is  under  the  direct 
control  of  major  program  managers  is  development  and  motivation  of  existing 
personnel.  According  to  interviewees,  skill  and  experience  development  through 
cross-training,  broad  exposure  to  new  challenges  and  different  varieties  of 
program  management,  and  attendance  at  training  courses  and  professional 
seminars  are  long-term  enablers  of  organizational  success.  Although  time 
consuming,  constant  improvement  of  workforce  skills  is  necessary  to  leverage 
new  technologies  and  to  enable  continuous  improvement.  Mentoring  by  senior 
personnel  is  another  aspect  of  development  that  can  have  good  results  (Fisher 
2005).  A  good  monitor  for  these  activities  is  the  development  and  review  of 
individual  leadership  development  plans  (ILDP).  These  are  contracts  between 
leaders  and  their  subordinates  that  identify  professional  goals  of  the  subordinates 
and  needs  of  the  organization.  They  then  focus  on  the  skills  that  the 
subordinates  need  to  attain  or  to  improve  their  own  standing  and  meet  the  needs 
of  the  organization.  ILDPs  provide  a  methodology  for  improving  the  skill  level  of 
the  employees,  and  a  baseline  for  the  manager  to  monitor  the  professional 
development  of  the  subordinates.  ILDPs  are  used  throughout  IWS  1.0  and  2.0, 
but  subsequent  monitoring  for  achievement  of  goals  has  been  individual 
manager  dependent  in  the  past. 

Retention  of  PEO  IWS’s  critical  human  capital  is  important  to  future 
success.  After  the  investment  of  time,  funds,  and  energy  in  training  and 
developing  personnel,  it  is  essential  that  these  valuable  employees  be  retained. 
Retention  is  enhanced  by  many  things  including  the  ability  of  employees  to 
improve  their  skills,  by  monetary  and  non-monetary  rewards,  and  promotions.  A 
good  leading  indicator  of  retention  can  be  employee  satisfaction.  Happy 
employees  tend  to  stay  on  the  job  longer,  and  tend  to  produce  better  work  results 
(Heskett,  Sasser,  and  Schlesinger  1997).  An  in-depth  analysis  of  contributors  to 
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employee  satisfaction,  through  interviews  or  questionnaire,  can  contribute 
significantly  to  the  managers  understanding  of  the  drivers  behind  retention. 

As  demonstrated  in  recent  efforts  to  focus  the  Navy  budget  on  the 
recapitalization  of  ships  and  aircraft,  reductions  in  manpower  can  be  viewed  as  a 
means  to  free  funds  for  higher  priority  needs.  Recent  reductions  in  contractor 
support  and  hiring  restrictions  within  NAVSEA  organizations  are  good  examples. 
While  that  may  be  the  reality  of  the  budget  situation,  accurate  human  capital 
metrics  are  needed  to  measure  the  impacts  of  those  decisions.  Much  like  a 
building  that  requires  consistent  investment  in  maintenance  to  sustain  itself  over 
time,  human  capital  needs  consistent  investment  of  time,  money,  and  energy  to 
continuously  improve  and  meet  organizational  strategic  goals. 

In  a  cost  cutting  environment,  improving  individual  employee 
productivity  is  essential  to  sustaining  or  improving  organizational  effectiveness 
while  maintaining  or  decreasing  manpower  costs.  There  is  a  finite  limit  to  which 
employee  costs  can  be  cut.  Improved  employee  commitment  from  a 
demonstration  by  IWS  1.0  and  2.0  to  employee  professional  growth  could  have  a 
positive  impact  of  productivity. 

Human  capital  as  an  outcome  aims  to  answer  an  important 
question,  “Do  we  have  the  right  number  of  properly  skilled  employees  in  the  right 
jobs  to  succeed  for  the  long-term?”  If  the  answer  to  that  question  is  yes,  then 
much  of  the  other  strategic  goals  of  IWS  1.0  and  2.0  should  be  attainable.  A 
large  part  of  ensuring  people  are  in  place  for  the  long-term  will  be  gauging  their 
commitment  to  the  organization.  It  is  not  enough  to  have  the  right  people  now, 
there  needs  to  be  some  confidence  that  they  will  be  there  in  the  future. 
Commitment  of  employees  to  the  organization,  along  with  their  satisfaction  and 
motivation  will  be  key  drivers  of  future  retention  (Fitz-enz  2000).  Additionally,  a 
plan  for  what  people  will  be  needed  in  the  long-term  will  be  required. 

2.  Programmatics 

It  was  apparent  from  the  interviews  that  three  primary  variables  have  been 
the  focus  of  performance  metrics  at  PEO  IWS  and  throughout  the  acquisition 
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community:  cost,  schedule  and  technical/performance.  All  but  one  of  the 
interview  subjects  named  these  three  variables  as  the  most  important  factors  to 
measure.  The  programmatics  perspective  contains  two  of  these  variables:  cost 
and  schedule. 

One  useful  method  for  analyzing  a  program’s  progress  towards 
completion  is  the  Earned  Value  Management  System  (EVMS).  EVMS  is  a 
management  tool  that  integrates  cost,  schedule  and  technical  performance  into 
one  tracking  system,  and  should  provide  excellent  indicators  of  progress  if 
reviewed  regularly.  An  EVMS  tracking  system  is  expensive  for  a  contractor  to 
set-up,  and  it  is  only  required  on  cost  or  incentive  contracts,  subcontracts,  intra¬ 
government  work  agreements,  and  other  agreements  valued  at  or  greater  than 
20  million  in  then-year  dollars. 

Earned  value  is  the  actual  physical  work  completed  on  a  project.  Cost 
performance  index  (CPI)  compares  the  budgeted  cost  of  work  performed 
(BCWP)  to  the  actual  cost  of  work  performed  (ACWP). 

Cost  Performance  Index  =  BCWP  /  ACWP 
Cost  variance  (CV)  is  determined  by  subtracting  ACWP  from  BCWP. 

Cost  Variance  =  BCWP  -  ACWP 

Schedule  performance  index  (SPI)  compares  the  BCWP  to  budgeted  cost  for 
work  scheduled  (BCWS). 

Schedule  Performance  Index  =  BCWP  /  BCWS 
Schedule  variance  (SV)  is  determined  by  subtracting  BCWS  from  BCWP. 

Schedule  Variance  =  BCWP  -  BCWS 

Together  these  indices  and  variances  indicate  actual  cost  and  schedule  relative 
to  plan.  With  variances,  a  positive  value  is  favorable.  With  the  indices,  a  value 
greater  than  one  is  favorable. 
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The  program  manager  variance  at  completion  (PM  VAC)  and  the 
contractor  variance  at  completion  (CONTR  VAC)  are  the  estimated  total  cost  at 
contract  completion  minus  total  budgeted  cost  at  contract  completion. 

Thresholds  are  set  above  and  below  an  established  baseline  to  act  as 
tripwires  to  cue  leadership  action.  They  indicate  to  anyone  who  looks  at  the  data 
whether  the  program  is  or  is  not  performing  to  plan,  whether  unfavorable  or 
favorable,  and  that  analysis  is  required  to  determine  the  cause. 

Even  when  EVMS  is  not  required,  the  basic  logic  should  still  be  followed. 
An  original  baseline  should  be  derived  from  which  cost  and  schedule  deviations 
can  be  observed  and  acted  upon,  if  required. 

a.  Cost 

Cost  of  a  weapon  system  is  an  outcome  of  contractor  performance 
and  program  management.  It  is  closely  tied  together  with  technical/performance 
and  schedule.  Trade-offs  between  cost,  schedule  and  performance  lead  to 
reductions  or  increases  in  their  nominal  values.  Indicators  of  cost  performance 
are  primarily  delivered  through  EVMS  and  close  communication  with  the 
contractor  and  Contracting  Officer’s  Representative  (COR).  The  use  of  EVMS  is 
growing  in  the  acquisition  community  with  its  use  required  at  lower  total  contract 
values  than  previously  mandated. 

b.  Schedule 

Schedule  is  influenced  by  many  variables  and  tracking  any  of  those 
should  be  indicative  of  future  schedule  slippage.  Funding,  cost,  technical 
performance,  employee  competency,  contractor  performance,  contract  design, 
and  T&E  can  all  influence  schedules.  Also,  changes  in  the  external  factors 
beyond  the  program  manager’s  control  can  have  a  large  impact  such  as  ship 
delivery  schedule  and  availability  of  ships  due  to  operational  “surge” 
requirements. 

c.  Test  and  Evaluation 

The  T&E  process  is  a  good  indicator  of  program  success.  Because 
of  the  iterative  process  of  T&E,  from  less  to  more  demanding  testing,  each 
separate  phase  provides  an  opportunity  to  evaluate  the  current  and  potential 
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success  of  the  program.  The  Defense  Acquisition  Guidebook  outlines 
Development,  Test,  and  Evaluation  best  practices  and  provides  some  ideas  for 
potential  leading  metrics  for  T&E  and  program  success.  These  metrics  are 
depicted  in  the  proposed  balanced  scorecard. 

d.  Risk  Management 

Managing  risk  is  one  of  the  essential  tasks  of  program 
management.  A  risk  management  strategy  is  determined,  usually  by  an 
integrated  team,  and  published  in  a  Risk  Management  Plan.  A  level  of  risk  can 
be  assigned  to  any  event  or  outcome,  but  it  is  of  primary  concern  in  program 
management  at  PEO  IWS  in  terms  of  cost,  schedule,  performance,  and  funding 
risk. 

One  common  method  to  determine  risk  is  by  a  risk  rating  matrix.  A 
generic  risk  rating  matrix  is  depicted  in  Figure  12.  Risk  events  are  decomposed 
into  likelihood  and  consequences.  Likelihood  is  stated  as  an  adjective  describing 
the  probability  of  occurrence,  either  remote,  unlikely,  likely,  highly  likely,  or  near 
certainty.  Consequences  are  described  by  potential  program  impact  if  a  risk 
event  were  to  occur,  either  minimal/none,  slight,  moderate,  significant,  or 
unacceptable.  A  risk  rating  is  then  assigned  by  plotting  the  risk  likelihood  on  the 
vertical  axis  of  a  chart  and  risk  consequences  on  the  horizontal  axis.  The 
intersection  of  the  likelihood  and  consequences  in  the  chart  are  the  risk  rating.  In 
Figure  12,  event  1  is  medium  risk  and  event  2  is  low  risk. 
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Figure  12.  Risk  Rating  Matrix  (From:  Naval  Postgraduate  School  2005) 

At  a  minimum,  a  subjective  risk  assessment  should  be  executed  by 
the  program  manager  based  on  experience  and  an  assessment  of  what  is  known 
about  program  variables  and  the  external  program  environment.  Objectivity 
would  be  preferred  over  subjectivity,  in  most  cases.  One  method  of  making  a 
more  objective  determination  is  by  assigning  specific  definitions  to  the  adjective 
ratings  for  likelihood  and  consequences  that  are  then  used  universally  throughout 
the  program.  The  benefits  might  include  better  trend  analysis  over  time  and  a 
reduction  in  bias  of  the  subjective  risk  rating.  The  risk  rating  can  be  used  by  the 
major  program  manager  to  track  risk  in  his  or  her  many  programs. 

Risk  must  be  tracked  systematically  during  the  acquisition  process 
(Naval  Postgraduate  School  2004).  Key  events  can  be  utilized  to  measure  risk. 
T&E  events  serve  as  good  points  to  monitor  and  reassess  risk.  Performance, 
cost,  and  schedule  progress  can  be  compared  against  a  pre-established  baseline 
to  evaluate  probability  of  attaining  ultimate  program  goals  within  predefined 
constraints. 
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The  Earned  Value  Management  provides  an  excellent  risk 
indicator.  At  any  given  point  in  system  development,  cost  and  schedule 
execution  can  be  compared  to  the  baseline  to  determine  current  status,  and 
future  status  can  be  forecast.  Risk  can  be  determined  from  current  progress 
relative  to  baseline. 

e.  Communication 

Communication  can  be  the  glue  that  holds  an  organization  together 
and  keeps  its  different  parts  moving  forward  together.  Interview  subjects 
emphasized  the  importance  of  communication  to  their  staying  abreast  of  the 
health  of  their  programs  and  in  building  relationships  which  were  essential  to 
good  program  management.  From  the  maps  it  is  evident  that  communication 
effectiveness  has  impacts  throughout  the  other  processes  and  outcomes.  As 
stated  earlier,  neither  IWS  1.0  nor  2.0  produces  the  systems  that  they  are 
responsible  for  providing  to  the  Fleet.  They  add  value  to  the  Navy  by  expertly 
managing  the  acquisition  and  support  of  those  systems.  To  provide  that  value 
they  must  coordinate  their  activities  both  internally  and  externally  and  gather  and 
disperse  information  to  all  stakeholders. 

Effective  communication  can  be  the  key  to  organizational  success. 
In  one  study,  researchers  concluded  that  the  benefits  obtained  from  quality 
internal  communications  included: 

•  improved  productivity, 

•  reduced  absenteeism, 

•  higher  quality  of  services  and  products, 

•  increased  levels  of  innovation, 

•  fewer  strikes,  and 

•  reduced  costs  (Clampitt  and  Downs  1993). 

Another  researcher  commented,  “The  manager  does  not  leave  meetings  or  hang 
up  the  telephone  in  order  to  get  back  to  work.  In  large  part,  communication  is  his 
or  her  work  (Mintzberg  1989).” 

It  can  be  very  difficult  to  monitor  the  inputs  to  and  process  of 

communication.  One  measure  of  an  input  to  communication  might  be  the 
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robustness  and  quality  of  the  information  exchange  network,  including 
databases,  email  and  telephones.  One  of  the  characteristics  of  a  good 
performance  metric  is  that  it  be  within  the  ability  of  the  manager  to  influence. 
With  the  advent  of  the  Navy’s  enterprise-wide  intranet  service,  the  flexibility  of  the 
major  program  manager  to  influence  communications  hardware  and  software  has 
been  curtailed,  thus  this  may  not  be  the  most  useful  metric. 

A  better  focus  for  a  measure  might  be  the  outcomes  of  the 
communication  process.  Many  of  the  outcomes  are  measurable  during  the  many 
reviews  that  an  acquisition  program  must  go  through  during  its  development  and 
ultimate  fielding.  These  incremental  reviews  give  indications  as  to  the  progress 
of  a  program.  It  is  evident  from  the  causal  performance  maps  that 
communication  can  cause  adverse  effects  to  that  progress.  One  way  to  prepare 
for  reviews  might  be  for  the  program  manager  to  ask  him  or  herself,  “What 
problems  could  poor  communication  cause  on  this  program,”  and  then  look  for 
those  problems  (Hargie  and  Tourish  2000).  If  the  program  is  not  making  desired 
progress  then  communications  might  be  one  of  the  contributing  factors.  The 
program  manager  must  make  a  detailed  analysis  of  the  effectiveness  of 
communication  at  that  time,  if  not  before.  A  leading  indicator  might  be  frequency 
of  interactions  with  stakeholders  in  a  process. 

One  periodic  and  effective  method  of  measuring  communications 

effectiveness  is  via  a  communications  audit.  The  audit  can  measure  or  identify 

quality  of  information  communicated,  quality  of  communication  relationships, 

organizational  communication  networks,  and  bottlenecks  in  existing  networks. 

The  audit  can  give  a  clearer  picture  about  what  is  happening  in  an  organization 

compared  to  what  leadership  might  think  is  happening  (Hurst  1991 ).  This  audit  is 

best  done  via  questionnaires  or  interviews.  The  least  time  intensive,  most 

anonymous,  and  best  method  for  trend  analysis  and  benchmark  comparisons 

tends  to  be  with  questionnaires  (Hargie  and  Tourish  2000).  The  audit  can  help 

identify  if  communications  are  flowing  fluidly  inside  and  outside  the  organization 

and  locate  points  that  are  acting  to  impede  communications  flow.  One  outcome 

of  this  process  might  be  to  identify  communications  training  that  can  help  the 
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organization  communicate  more  effectively.  Other  benefits  might  be  to  recognize 
an  inefficient  organizational  structure  or  to  identify  needed  communications 
infrastructure  improvements. 


Another  more  subjective  method  could  be  to  emphasize  the 
importance  of  communication  to  employees  and  subsequent  review  of  their 
performance  in  that  area  during  annual  individual  performance  reviews.  What 
gets  emphasized  daily  and  during  reviews  is  hopefully  where  employees  will 
maximize  their  efforts.  Additionally,  some  amount  of  monetary  awards  can  be 
dedicated  to  rewarding  employees  who  display  effective  communication  as 
measured  in  communication  audits,  or  other  methods. 

Communication  is  the  basis  for  relationships  and  international 
cooperation.  The  amount  of  cooperative  weapon  systems  development  with 
allies  could  be  an  indicator  of  communication’s  effectiveness,  assuming  an 
ongoing  effort  for  international  engagement  exists. 

f.  Teamwork 

Teamwork  is  especially  important  for  IWS  1.0  because  of  its  focus 
on  integrated  combat  systems.  It  is  very  difficult  to  build  an  integrated  system 
without  sound  teamwork.  IWS  1 .0  integrates  the  work  of  multiple  major 
programs  into  a  fully  combat-capable  system.  The  AEGIS  radar  is  a  good 
example.  IWS  1.0  handles  the  combat  system  while  IWS  2.0  handles  the  radar 
hardware  portion  of  the  system.  Without  close  coordination  and  teamwork,  the 
final  combat  system  will  not  function  as  well  as  envisioned,  if  at  all.  The 
composite  maps  for  both  IWS  1 .0  and  2.0  show  the  ultimate  affects  of  teamwork 
on  the  outcomes  of  the  organizations.  For  IWS  1.0  it  influences  all  but 
development  of  human  capital,  and  for  IWS  2.0  it  has  equally  significant  impacts. 

Like  communication,  teamwork  can  be  difficult  to  objectively  and 
quantifiably  measure  at  the  input  or  process  stage.  One  useful  point  of 
measurement  might  be  the  degree  of  stakeholder  participation  in  regular  program 
meetings.  Those  stakeholders  would  include  but  are  not  limited  to  other  program 
managers  that  are  involved  in  development  of  a  particular  combat  system, 
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weapon  system  contractors,  resource  sponsors,  T&E  personnel,  and  Fleet 
representatives.  Teamwork  is  the  key  to  the  IPPD  process.  Strong  teamwork 
and  a  multi-disciplinary  approach,  started  at  the  beginning  of  a  program’s 
conception  and  development,  should  improve  chances  of  success  and  decrease 
total  life  cycle  cost  of  a  weapon  system  (Department  of  Defense  2005a). 

How  well  integrated  a  program  is  should  be  evident  from  the 
previously  mentioned  communication  audits.  It  is  difficult  to  be  integrated  without 
effective  and  regular  communication.  If  the  communication  audit  indicated  that  a 
key  member  of  a  system  development  team  was  working  in  isolation  that  might 
be  a  good  indicator  of  future  problems  and  an  area  for  management  action. 

Planning  reviews  and  preparations  for  them  are  good  points  for 
manager  evaluation  of  the  degree  of  teamwork  on  a  program.  The  manager 
should  be  able  to  gauge  the  degree  of  internal  and  external  coordination  by  the 
review.  One  of  the  key  IPPD  tenets  is  multidisciplinary  teamwork  through 
Integrated  Product  Teams  (Department  of  Defense  2005a).  A  determination  of 
how  well  the  IPPD  principles  are  being  followed  could  be  one  of  the  take-aways 
from  a  review. 

Sharing  of  award  incentives  could  motivate  teamwork  amongst  both 
PEO  IWS  employees  and  contractors.  In  order  for  anyone  to  collect  award  fees 
or  personal  reward  incentives,  the  entire  team  must  meet  its  goals.  This  could 
hopefully  foster  teamwork  since  all  would  have  an  incentive  to  help  each  other 
succeed. 

T&E  can  provide  a  lagging  indicator  of  how  well  integrated  a 
program  is.  If  the  separate  parts  that  make  up  a  system  have  not  been  designed 
and  assembled  in  a  coordinated  fashion,  with  good  teamwork  between  all 
concerned,  then  the  results  of  T&E  will  most  likely  be  below  expectations. 

g.  Contractor  Performance 

By  the  nature  of  the  business,  contractor  performance  is  critical  to 
the  success  of  any  acquisition  program.  From  the  causal  maps  it  is  relatively 
easy  to  see  that  contractor  performance  impacts  technical/performance,  cost  and 
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schedule.  Regardless  of  the  expertise  of  the  program  manager,  if  the  contractor 
does  not  adequately  perform,  despite  the  program  manager’s  best  efforts,  the 
program  will  not  succeed.  That  said,  the  best  way  for  the  program  manager  to 
specifically  monitor  contractor  performance  is  through  frequent  and  precise 
communication,  and  a  close  relationship  with  the  COR  and  the  contractor  team. 
Frequent  and  honest  assessments  are  essential,  as  is  an  accurate  EVMS 
appraisal  at  regular  intervals. 

Contractor  performance  must  be  evaluated  for  its  impact  on  system 
cost,  schedule,  and  technical/performance.  EVMS  allows  that  evaluation.  EVMS 
data  is  frequently  at  least  30  days  out  of  date  by  the  time  it  reaches  a  program 
manager.  Close  coordination  with  the  contractor  and  the  COR  allows  the 
program  manager  to  get  up  to  date  information  when  required. 

When  used  properly,  EVM  allows  government  and  contractor 
program  managers  visibility  into  technical,  cost  and  schedule 
planning,  performance  and  progress  on  their  contracts.  This 
visibility  provides  not  only  insight  to  contract  performance,  but 
provides  data  points  to  statistically  estimate  completion  costs 

(Department  of  Defense  2005b). 

EVMS  has  been  a  part  of  program  management  for  37  years  and  is 
well  understood  in  the  acquisition  community.  The  recent  reduction  in  threshold 
for  implementation  of  EVMS  from  certain  contracts  valued  at  greater  than  73 
million  dollars  to  those  valued  at  greater  than  20  million  dollars  appears  to  be 
indicative  of  its  perceived  value  as  a  tracking  tool  for  program  health. 
h.  Contract  Design 

Contract  design  can  have  a  large  impact  on  contractor 
performance,  cost  and  schedule.  If  insufficient  risk  is  borne  by  the  contractor, 
there  may  be  a  reduced  incentive  to  deliver  on  time  and  on  budget.  Specificity  of 
terms  within  the  contract  seems  to  be  of  significant  importance.  Additionally, 
team  incentives  might  prove  useful  in  generating  teamwork  among  contractors. 

/'.  Planning 

The  old  saying  goes,  “Prior  planning  prevents  poor  performance.” 
That  is  as  true  in  acquisition  as  in  other  endeavors.  An  integrated  and 
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comprehensive  planning  process  can  lead  to  excellent  results.  Time  invested  at 
the  beginning  of  the  process  should  pay  dividends  at  the  end  by  minimizing 
rework.  The  review  and  decision  point  processes  within  the  acquisition 
framework  provide  adequate  opportunities  to  gauge  the  robustness  of  planning 
and  strategy  development.  A  lagging  indicator  might  be  the  historic  accuracy  of 
planned  cost  and  schedule  performance  for  IWS  1.0  and  2.0.  Continuously 
improving  accuracy  might  be  indicative  of  accurate  planning  and  forecasting. 

j.  Contractor  Past  Performance 

Contractor  past  performance  can  be  an  indicator  of  future 
performance  and  can  act  as  a  leading  indicator  of  success.  It  can  be  evaluated 
using  the  Contractor  Performance  Assessment  Reporting  System  (CPARS).  A 
possible  metric  might  relate  the  total  number  of  contracts  within  a  major  program 
area  to  number  of  contracts  with  contractors  that  have  received  favorable 
average  past  performance  ratings. 

k.  Technology  Maturity 

The  maturity  of  required  technology  can  be  accurately  gauged  via 
technology  readiness  assessments.  It  is  the  primary  initial  indicator  of  risk.  If  the 
system  requires  yet  to  be  developed  technology,  that  would  be  indicative  of  high 
risk.  That  risk  will  ultimately  help  determine  contract  design  and  impact 
acquisition  strategy. 

/.  IWS  2.0  Specific  CPVs 

(1)  International  Cooperation.  International  cooperation  was 
deemed  important  in  IWS  2.0  by  subject  F.  It  is  enhanced  through  strong 
relationships  and  regular  communication.  It  acts  to  mitigate  risk  and  stimulate 
new  program  creation  by  allowing  access  to  technology  that  might  be  superior  to 
that  available  from  U.S.  defense  contractors.  Even  if  not  superior,  the  different 
perspectives  regarding  system  design  and  development  found  amongst  our  allies 
might  offer  innovative  solutions. 

(2)  EW  and  Radar  Integration.  EW  and  radar  integration  is 
a  goal  for  IWS  2.0  focused  on  getting  more  combat  effectiveness  by  maximizing 
the  benefits  that  both  EW  and  radar  can  bring  to  the  warfighter.  From  the 
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interviews  it  was  evident  that  the  effort  needs  to  focus  on  getting  the  warfighter  to 
have  confidence  in  the  EW  system’s  ability  to  detect  and  deceive  a  threat.  That 
goal  can  be  achieved  through  T&E  and  Fleet  exercises  and  subsequent  active 
communication  of  those  results  to  the  Fleet  in  order  to  engender  confidence. 

(3)  New  Project  Creation.  New  project  creation  for  its  own 
sake  is  not  the  goal  that  IWS  2.0  is  after.  The  interview  subjects  indicated  that  it 
is  only  in  improving  Fleet  readiness  in  a  cost  effective  manner  that  new  projects 
make  sense.  The  need  for  and  creation  of  new  projects  is  influenced  by 
international  cooperation  and  innovative  employee  ideas.  IWS  2.0  has  access  to 
knowledge  concerning  cutting-edge  technology  and  allied  weapon  development 
efforts.  By  maintaining  and  enhancing  ties  to  allies  and  industry,  they  can  derive 
innovative  ways  to  improve  Fleet  weapon  systems. 

3.  Fleet  Readiness 

a.  Technical/Performance 

Operating  as  designed  and  meeting  key  performance  parameters 
(KPP)  as  laid  out  in  requirements  are  essential  outcomes  in  weapons  acquisition. 
All  six  interview  subjects  identified  this  along  with  cost  and  schedule  as  key 
outcomes  that  defined  success.  Performance  is  primarily  evaluated  via  T&E. 
System  performance  is  compared  against  key  performance  parameters  and 
specific  test  criteria  to  evaluate  system  readiness  to  proceed  to  the  next  level  of 
testing  and  ultimately  on  to  Fleet  introduction. 

b.  Supportability 

IWS  2.0  personnel  specifically  identified  supportability  as  an  area  of 
concern  and  a  critical  performance  variable.  It  seems  that  financial  support  for 
training  and  technical  manuals  is  one  of  the  easier  items  to  reduce  if  there  are 
cost  overruns  on  a  program.  Supportability  is  more  than  just  technical  manuals 
and  also  includes  a  good  plan  to  keep  the  systems  operating  at  full  potential. 
IWS  1 .0  interview  subjects  included  supportability  as  a  subset  of 
technical/performance.  The  metrics  in  the  balanced  scorecard  are  germane  to 
both  major  programs. 
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c.  Customer  Satisfaction 

Customer  satisfaction  is  the  ultimate  test  of  the  acquisition  system. 
If  the  Fleet  gets  the  weapon  systems  it  needs  to  fight  and  win,  and  is  satisfied 
with  the  reliability,  usability,  supportability,  effectiveness,  and  cost,  then  PEO 
IWS  1 .0  and  2.0  will  most  likely  consider  their  efforts  a  success.  Communication 
plays  a  role  in  satisfaction.  The  ultimate  user  needs  to  have  an  expectation 
regarding  a  new  system.  System  function  will  be  judged  against  that 
expectation.  If  there  is  no  communication  between  the  acquisition  community 
and  the  Fleet  concerning  a  new  system’s  use  and  capabilities,  then  there  is  likely 
to  be  a  misguided  expectation  as  well  as  lower  acceptance  of  the  new  system. 
This  expectation  can  also  be  influenced  by  the  amount  of  involvement  in  system 
planning  by  Fleet  personnel.  The  more  involvement,  the  more  likely  that  Fleet 
needs  will  be  identified  and  incorporated,  positively  impacting  ultimate 
satisfaction. 

Another  useful  tool  to  measure  customer  satisfaction  would  be  a 
customer  satisfaction  survey.  Additionally,  weapon  systems  instructors  and  field 
engineering  activities  that  respond  to  system  casualty  reports  (CASREPS)  would 
be  a  useful  and  knowledgeable  survey  audience  to  get  a  feel  for  Fleet 
satisfaction  with  IWS  1 .0  and  2.0  efforts. 

4.  Business 

a.  Financial  Management/Funding 

Money,  in  the  form  of  funding,  provides  the  grease  that  keeps  the 
acquisition  machine  running.  Sufficiency  of  funding  and  appropriate 
management  of  those  funds  impact  cost,  schedule  and  technical/performance. 
Financial  management  is  impacted  by  the  expertise  of  the  personnel  that  are 
managing  the  funds,  effectiveness  of  communication,  teamwork,  and  planning. 
Some  key  metrics  for  financial  management  effectiveness  are  listed  below. 

Maintenance  of  funding  across  the  FYDP  is  vital  to  program 
schedule  and  cost  maintenance,  as  well  as  meeting  technical/performance 
requirements.  Program  strategies  and  contract  execution  are  all  dependent  on 
maintaining  planned  funding.  The  best  metric  to  use  for  monitoring  funding 
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overtime  is  tracking  rescissions  and  reductions  to  a  budget.  The  best  leading 
indicator  of  threats  to  program  outcomes  is  the  difference  between  estimated 
funding  required  and  funding  currently  programmed. 

B.  BALANCED  SCORECARD 

The  performance  metrics  suggested  in  the  Proposed  Balanced 
Scorecards  (Tables  3-5)  may  be  currently  useful  for  IWS  1.0  and  2.0,  but  they 
need  to  be  flexible  over  time  for  multiple  reasons.  First,  as  the  strategic 
environment  changes,  IWS  1 .0  and  2.0  will  need  to  change  and  adjust  in  order  to 
continue  to  improve  and  succeed.  The  performance  metrics  may  need  to  change 
as  well,  since  they  should  be  a  reflection  of  the  organizational  strategy. 

Second,  organizations  learn  to  maximize  performance  relative  to 
metrics.  That  was  one  reason  to  communicate  metrics  throughout  an 
organization  ~  to  influence  employee  behavior.  While  maximizing  what  is 
measured,  other  parts  of  the  strategy  may  be  neglected,  or  other  measurable 
aspects  of  the  CPVs  and  outcomes  may  be  ignored. 

Third,  performance  metrics  may  cause  unintended  dysfunctional 
behavior  by  employees  trying  to  maximize  performance  relative  to  the  metrics. 
For  those  reasons  and  probably  more,  the  performance  metrics  should  be  refined 
regularly  to  ensure  they  are  still  capturing  relevant  information  for  achieving  the 
organizations  strategic  goals.  The  result  of  that  review  is  hopefully  a  zero  sum 
evolution.  If  any  measures  are  added,  others  should  be  taken  out,  so  that  a  sort 
of  metrics  overload  does  not  take  hold;  measures  should  be  prioritized  for  value 
toward  achieving  the  strategy. 
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Table  3.  IWS  1.0  and  2.0  Proposed  Balance  Scorecard  -  Personnel 
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Table  4.  IWS  1.0  and  2.0  Proposed  Balance  Scorecard  -  Programmatics 
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Table  5.  IWS  1.0  and  2.0  Proposed  Balance  Scorecard  -  Fleet  Readiness  and 

Business 
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V.  CONCLUSIONS  AND  RECOMMENDATIONS 


A.  INTRODUCTION 

There  is  an  obvious  need  for  performance  measurement  in  Navy 
commands.  It  is  required  by  statute  and  desirable  from  a  leadership  standpoint. 
This  thesis  has  attempted  to  demonstrate  a  methodology  for  arriving  at 
performance  metrics,  in  this  case  for  IWS  1 .0  and  2.0.  First,  performance  drivers 
were  captured  for  IWS  1.0  and  2.0  via  interviews  with  key  leaders  in  those  two 
major  programs.  Those  interviews  were  transformed  into  causal  performance 
maps  which  depicted  the  interplay  of  the  drivers  and  the  outcomes  they 
influence.  Performance  metrics  were  then  derived  for  the  drivers  and  outcomes 
and  arranged  in  a  balanced  scorecard  format.  The  scorecard  may  be  useful  to 
the  major  program  managers  in  monitoring  the  progress  of  their  organizations  at 
achieving  strategic  goals.  Additionally,  the  metrics  should  enhance 
understanding  of  strategic  direction  by  the  rank  and  file  of  IWS  1 .0  and  2.0  if  they 
are  openly  discussed  and  analyzed. 


B.  BALANCED  PERSPECTIVE 

It  was  evident  from  the  interviews  and  the  causal  performance  maps  that 
there  are  many  diverse  factors  that  impact  strategic  organizational  success  for 
IWS  1.0  and  2.0.  Those  factors  are  under  the  influence  of  the  major  program 
managers  to  varying  degrees.  The  performance  metrics  must  attempt  to 
measure  across  the  four  balanced  scorecard  perspectives  for  IWS  1 .0  and  2.0, 
not  just  programmatics  and  Fleet  readiness,  if  they  are  to  adequately  support 
progress  toward  long-term  goal  achievement. 


C.  PEO  IWS  IS  ON  THE  RIGHT  TRACK 

PEO  IWS  recognized  at  the  outset  of  FY  2005  that  it  needed  to  improve  its 
human  capital  strategy,  horizontal  integration,  and  performance  metrics. 
Currently,  the  command  has  an  active  human  capital  strategy  team  that  is  further 
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developing  and  refining  its  human  capital  strategy.  PEO  IWS  has  also  created  a 
horizontal  integration  team  to  improve  organizational  efficiency  by  identifying 
redundant  functions  as  well  as  areas  that  they  can  leverage  organizational 
synergy  to  improve  productivity.  A  performance  metrics  integrated  product  team 
is  also  actively  pursuing  PEO  level  metrics  to  better  focus  the  organization’s 
energy  and  measure  results.  They  have  leveraged  expert  facilitators  to  help 
achieve  success  in  all  three  areas.  Interestingly,  this  thesis  recognized  similar 
areas  for  improvement,  but  arrived  at  that  conclusion  by  a  completely  different 
methodology.  The  three  main  areas  for  metrics  improvement  recognized  by  this 
research  are  human  capital,  teamwork  (integration),  and  communication.  These 
can  also  be  the  hardest  areas  to  measure  because  of  the  difficulty  in  objectively 
quantifying  performance  in  those  areas,  particularly  in  teamwork  and 
communication. 

As  part  of  NAVSEA,  PEO  IWS  was  involved  in  an  assessment  by 
GENESYS  Corporation  regarding  organizational  alignment.  The  survey  gave 
insight  to  the  major  program  managers  in  six  areas:  Mission,  Service,  People, 
Processes,  SEA  POWER  21  and  Transformation  Framework.  Each  of  the  six 
factors  could  be  further  divided  for  analysis.  People  could  be  divided  into 
Supervision;  NAVSEA  Leadership  (Flag/SES);  Job  Skills  &  Knowledge; 
Performance  Accountability;  and  Motivational  Energy.  The  information  was 
presented  in  such  a  way  that  users  could  determine  how  well  aligned  the 
employee  and  leadership  perspectives  were  within  the  command.  If  it  was 
apparent  that  there  was  poor  alignment  with  leadership’s  goals  then  that  might 
indicate  the  need  for  deeper  investigation  to  determine  the  reasons  for 
misalignment  and  an  appropriate  courses  of  action.  At  a  minimum  the  survey 
might  be  used  for  trend  analysis  by  the  major  program  managers.  It  could 
probably  prove  a  good  indicator  of  communication  and  teamwork  effectiveness, 
as  well. 
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D.  VALUE  OF  METHODOLOGY  TO  THE  NAVY 

The  methodology  used  in  this  thesis,  ethnographic  interview  and  causal 
map  creation,  are  an  effective  way  to  derive  CPVs  for  an  organization,  and  can 
provide  a  valuable  starting  point  for  a  performance  metric  working  group.  As 
stated  earlier,  the  best  and  most  viable  performance  metrics  usually  come  from  a 
group  effort  within  an  organization.  Selection  of  interview  subjects  is  important 
so  that  the  mapping  process  leverages  tacit  knowledge  of  individuals  who  know 
how  the  organization  truly  functions  at  both  a  strategic  and  tactical  level.  The 
presentation  of  the  map  to  a  metrics  working  group  would  be  an  excellent  place 
to  start  that  group’s  discussions.  The  group’s  time  could  be  more  efficiently 
spent  working  from  the  CPVs  and  outcomes  identified  by  the  causal  performance 
map,  or  from  a  balanced  scorecard  or  similar  display  of  potential  metrics.  Large 
groups  sometimes  have  difficulty  agreeing  on  an  initial  framework  and  direction. 
The  work  done  in  arriving  at  CPVs  should  provide  an  initial  framework  for 
working  group  efforts  and  jump  start  the  group  toward  a  successful  outcome. 

The  use  of  the  causal  map  to  derive  performance  metrics  without  the  input 
of  a  stakeholder  group  is  not  recommended.  The  resulting  metrics  are  potentially 
subjective  and  run  a  very  great  risk  of  missing  an  important  element  of  the  CPVs 
and  outcomes. 

E.  POTENTIAL  FOR  FURTHER  RESEARCH 

Analysis  of  the  success  of  a  performance  measurement  system  in  a 
command  that  has  recently  reinvigorated  its  metrics  might  prove  useful.  There 
are  currently  many  Navy  commands  revitalizing  their  measurement  systems,  just 
like  PEO  IWS.  These  commands  provide  opportunity  for  future  analysis. 

•  Which  methods  for  deriving  metrics  work  best? 

•  Which  specific  metrics  and  types  of  metrics  were  least  expensive  to 
implement  and  track,  both  in  time  and  money? 

•  What  types  of  metrics  were  most  useful  to  leaders  and  managers? 

•  Is  there  a  measurable  difference  in  performance  before  and  after 
the  implementation  of  the  new  performance  metrics? 
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The  Navy  is  operating  in  a  long-term  environment  of  fiscal  constraint. 
There  will  continue  to  be  a  push  for  improved  efficiency  and  effectiveness.  Any 
research  that  can  help  commands  make  measurable  improvements  will  be  useful 
to  Navy  leaders. 
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